Num_workers in DataLoader will increase memory usage?

flyaway: <p>I am using DataLoader to load my training data.<br>
According to the document, we can set <code>num_workers</code> to set the number of subprocess to speed up the loading process.<br>
However, when I use it like this:</p>
<pre><code class="lang-auto">dataloader = DataLoader(dataset, batch_size=args.batch_size,
shuffle=True, drop_last=False, num_workers=10,
collate_fn=dataset.collate_fn)
</code></pre>
<p>I found the memory usage keep growing, which is not happening when I set <code>num_worker=0</code>.<br>
Is this supposed behavior ? I have limited memory resources, so I don’t want the memory usage keeps growing.</p>
<p>P.S. My code is running on the GPU, every time I move a batch of data from cpu to gpu.</p>

SimonW: <p>how does your dataset code look like?</p>

ngxbac: <p>I have the same problem as him. Here is my topic</p><aside class="quote quote-modified" data-post="1" data-topic="28759">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ngxbac/48/7410_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/memory-fragmentation/28759">Memory fragmentation</a> 
  </div>
  <blockquote>
    Hello, 
I am dealing with memory fragment when using dataloader. 
Following is my __getitem__ function: 
    def __getitem__(self, idx):

        # Get token
        token = self.tokens[idx]

        # Get drawing
        drawing = self.drawings[idx]
        stroke = stroke_data(drawing).astype(np.float32)
        # stroke = np.expand_dims(stroke, axis=0)
        stroke = np.transpose(stroke, (1, 0)).astype(np.float32)

        drawing = eval(drawing)
        label = self.labels[idx]

        # …
  </blockquote>
</aside>

<p>Please help us</p>

lelouedec: <p>I think that when using multiple workers, each of them loads its data on the cpu before sending it to gpu. Which makes the amount of data in memory bigger. Also it creates a new process per worker.</p>

ngxbac: <p>I found other similar question like this. Here is one of example.<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/13246" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <a href="https://github.com/bfreskura" rel="nofollow noopener">
<img src="https://avatars1.githubusercontent.com/u/9371112?v=2&amp;s=96" class="thumbnail onebox-avatar" width="60" height="60">
</a>

<h4><a href="https://github.com/pytorch/pytorch/issues/13246" target="_blank" rel="nofollow noopener">Issue: CPU memory gradually leaks when num_workers &gt; 0 in the DataLoader</a></h4>

<div class="date" style="margin-top:10px;">
	<div class="user" style="margin-top:10px;">
	opened by <a href="https://github.com/bfreskura" target="_blank" rel="nofollow noopener">bfreskura</a>
	on <a href="https://github.com/pytorch/pytorch/issues/13246" target="_blank" rel="nofollow noopener">2018-10-29</a>
	</div>
	<div class="user">
	</div>
</div>

<pre class="content" style="white-space: pre-wrap;">🐛 Bug
CPU memory will leak if the DataLoader num_workers &gt; 0.
To Reproduce
Run the following snippet:
class DataIter(Dataset):
 def __init__(self):
 path = "path/to/data"
...</pre>

<div class="labels">
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">medium priority</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
In this thread, they said it is fixed recent day at:<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/13243" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <a href="https://github.com/shhshn" rel="nofollow noopener">
<img src="https://avatars1.githubusercontent.com/u/15028546?v=2&amp;s=96" class="thumbnail onebox-avatar" width="60" height="60">
</a>

<h4><a href="https://github.com/pytorch/pytorch/issues/13243" target="_blank" rel="nofollow noopener">Issue: Potential memory leak in Tensor.size()</a></h4>

<div class="date" style="margin-top:10px;">
	<div class="user" style="margin-top:10px;">
	opened by <a href="https://github.com/shhshn" target="_blank" rel="nofollow noopener">shhshn</a>
	on <a href="https://github.com/pytorch/pytorch/issues/13243" target="_blank" rel="nofollow noopener">2018-10-29</a>
	</div>
	<div class="user">
	closed by <a href="https://github.com/facebook-github-bot" target="_blank" rel="nofollow noopener">facebook-github-bot</a>
	on <a href="https://github.com/pytorch/pytorch/issues/13243" target="_blank" rel="nofollow noopener">2018-10-30</a>
	</div>
</div>

<pre class="content" style="white-space: pre-wrap;">🐛 Bug
Tensor.size() leaks 28 bytes of memory (similar to the previously resolved issue of #1963)
To Reproduce
import torch, tracemalloc; tracemalloc.start()
def main():
 def...</pre>

<div class="labels">
 	<span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">high priority</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>I am fixing my local torch’s code. Hopefully I help</p>

VISHNU_SUBRAMANIAN: <aside class="quote no-group" data-username="ngxbac" data-post="5" data-topic="28522">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ngxbac/48/7410_2.png" class="avatar"> ngxbac:</div>
<blockquote>
<p>Issue: Potential memory leak in Tensor.size()</p>
</blockquote>
</aside>
<p>I am facing this issue even with the updated PyTorch nightly version. The Dataloder memory usage continuously increases until it runs of memory. My Dataset size is 26GB when initialized, it contains an ndarray from which I return an element based on index value. After running on 10% of data it ends up using another 30+GB of ram and 40GB + swap space. I tried upgrading the PyTorch with the latest nightly version. Tried both on python3.6 and Python3.7, but the issue persists.</p>
<p>Thanks,<br>
Vishnu</p>

Liang: <p>Now, for PyTorch 1.2.0, multiple workers don’t make multiple memory copy of dataloader object. But still  such problems some times. I find that setting <code>num_worker &lt; physical cpu kernels</code> works fine.</p>

suseyang: <p>How do you implement your dataset?<br>
I am using PyTorch 1.5.0, but it still make memory copy, based on my observation.</p>

Liang: <p>I just load the whole dataset in the init() of DataLoader, others are just as normal. Be sure to make <code>num_worker &lt; physical cpu </code>. I don’t know why, but it works well when your memory is limited. E.g., my dataset is about 30 GB, I can run on a two-GPU machine with 80 GB memory. Num_worker and physical cpu are 10 and 12 respectively.</p>

maplewizard: <p>It is a feature of Python. You may encounter this problem when using native Python structures like map or list. I have developed a <strong>NEW TOOL</strong> called <code>cstl</code> (<a href="https://github.com/fuzihaofzh/cstl" class="inline-onebox" rel="noopener nofollow ugc">GitHub - fuzihaofzh/cstl: The C++ Standard Template Library (STL) for Python.</a>) by wrapping C++ STL containers to solve this issue. It supports multiple types including nested maps and nested lists which the numpy and pytorch do not support.<br>
Here is a simple example showing how it solves the problem:</p>
<pre><code class="lang-python">from torch.utils.data import Dataset, DataLoader
import numpy as np
import torch
import copy
import sys
import cstl
from tqdm.auto import tqdm


class DataIter(Dataset):
    def __init__(self):
        cnt = 24000000
        self.cnt = cnt
        #self.data = np.array([x for x in range(cnt)]) # Good
        #self.data = [x for x in range(cnt)] #Leaky
        #self.data = cstl.MapIntInt({i : i for i in range(24000000)})# Good
        self.data = cstl.VecInt(range(24000000)) # Good

        
    def __len__(self):
        return self.cnt

    def __getitem__(self, idx):
        data = self.data[idx]
        data = np.array([int(data)], dtype=np.int64)
        return torch.tensor(data)

train_data = DataIter()
train_loader = DataLoader(train_data, batch_size=300,
                          shuffle=True,
                          drop_last=True,
                          pin_memory=False,
                          num_workers=18)

for i, item in tqdm(enumerate(train_loader)):
    torch.cuda.empty_cache()
    if i % 1000 == 0:
        print(i)
</code></pre>

AKACHI-4: <p>Hey guyz, anyone find the root cause behind this that why memory usage rise highly with increasing num_workers, I did some research and found that memory getting in high due to tensors we are keeping in cpu memory that transferred to gpu in process, but it somehow release them after sometime, m not sure how it does that . ?</p>