CPU RAM usage increasing for every epoch

username: kunasiramesh
cooked: <p>Hello,<br>
I’m running into troubles while training a CAE(Convolutional Auto Encoder) model. I defined my own dataset class as follows:</p>
<pre><code class="lang-python">def make_dataset(dir, class_to_idx, extensions):
    label_list = []
    input_list = []
    dir = os.path.expanduser(dir)
    for target in sorted(class_to_idx.keys()):
        d = os.path.join(dir, target)
        if not os.path.isdir(d):
            continue

        for root, _, fnames in sorted(os.walk(d)):
            for fname in sorted(fnames):
                if has_file_allowed_extension(fname, extensions):        
                    if target == 'Source':
                        if fname.endswith('.wav'):
                            path = os.path.join(root, fname)
                            item = (path)
                            label_list.append(item)
                    else:
                        if fname.endswith('.wav'):
                            path = os.path.join(root, fname)
                            item = (path)
                            input_list.append(item)
    return input_list, label_list



class msd_dataset(data.Dataset):

    def __init__(self, root, extensions, train=True, input_transform=None, target_transform=None):

        self.extensions = extensions
        self.train = train

        #Join the Train/Test to the existing path if user specifies Train=True/False respectively.
        if self.train:
            self.root = os.path.join(root, 'Train')
        else:
            self.root = os.path.join(root, 'Test')
          
        classes, class_to_idx = self._find_classes(self.root)
       
        samples = make_dataset(self.root, class_to_idx, extensions)
        #Sanity check 
        if (len(samples[0]) or len(samples[1])) == 0:
            raise(RuntimeError("Found 0 files in subfolders of: " + self.root + "\n"

       self.classes = classes
        self.samples = samples

        self.input_transform = input_transform
        self.target_transform = target_transform

    def _find_classes(self, dir):
       
        if sys.version_info &gt;= (3, 5):
            # Faster and available in Python 3.5 and above
            classes = [d.name for d in os.scandir(dir) if d.is_dir()]
        else:
            classes = [d for d in os.listdir(dir) if os.path.isdir(os.path.join(dir, d))]
        classes.sort()
        class_to_idx = {classes[i]: i for i in range(len(classes))}
        return classes, class_to_idx

def __getitem__(self, index):
        path, target = self.samples
        #Loading Audio with librosa 
        input_mixture, sr1 = librosa.load(path[index], sr=44100, mono=False)
        label_source, sr2 = librosa.load(target[index], sr=44100, mono=False)
        if self.input_transform is not None:
            input_mixture = input_mixture.reshape(input_mixture.shape[0], 1, input_mixture.shape[1])
            input_mixture = torch.from_numpy(input_mixture)
            input_mixture = self.input_transform(input_mixture)
        if self.target_transform is not None:
            label_source = label_source.reshape(label_source.shape[0], 1, label_source.shape[1])
            label_source = torch.from_numpy(label_source)
            label_source = self.target_transform(label_source)
        return input_mixture, label_source

    def __len__(self):
        return len(self.samples[0])
</code></pre>
<p>How can I fix this?</p>

username: kunasiramesh
cooked: <p>I have seen some post regarding this memory issue. But I didn’t any solution. I am using gc.collect() after every epoch.</p>

username: ptrblck
cooked: <p>Do you get any error message or do you have another issue regarding this <code>Dataset</code>?</p>
<p>PS: I’ve formatted your code. You can add code using three backticks `.</p>

username: kunasiramesh
cooked: <p>I don’t have any issues with this code. But cpu RAM is increasing continuously while training. I am loading one by one WAV file with dataloader</p>

username: Gkv
cooked: <p>Hi,<br>
I am also facing similar issues. In my case, I have all the features in disk as .pt file. I am loading it into RAM as some global variables and using in the dataloader by indexing it. The problem is, CPU RAM is increasing every epoch and after some epochs the process got killed by the OS. My question is, I already loaded the features into the memory, in the dataloader i am just using it, how  this is consuming extra memory?<br>
Thanks</p>

username: ptrblck
cooked: <p><a class="mention" href="/u/kunasiramesh">@kunasiramesh</a>, <a class="mention" href="/u/gkv">@Gkv</a> The memory issue might be related to the training procedure or another part of the code.<br>
Could you post the code so that we can have a look?<br>
Usually the computation graph is unintentionally stored somewhere, e.g. by using <code>losses += loss</code> instead of <code>losses += loss.item()</code>.</p>

username: InnovArul
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>: just to understand a bit more about OOM issues, if the computation graphs are stored unintentionally (assuming GPUs are used for training), it should lead to GPU out of memory. Am I right?</p>

username: ptrblck
cooked: <p>Exactly! However, if you are using the model on CPU, your CPU RAM will be filled.<br>
Maybe that’s the case here.</p>

username: Gkv
cooked: <p>Hi,<br>
I am posting the main part of the code here.</p>
<pre><code class="lang-auto">
#new
global full_train_set1
global full_train_set2
global val_set1
global val_set2
full_train_set1=torch.load('path to the featueres1 dict.pt')
full_train_set2=torch.load('path to the featueres2 dict.pt') 

val_set1=torch.load('path to the val features1 dict.pt')
val_set2=torch.load('path to the val feature2 dict.pt')
class MyCustomDataset(Dataset):
    def __init__(self,ids,train):
        self.names = ids
        self.train=train
    def __getitem__(self, index):
        my_id=self.names[int(index)]
        if self.train:
        	feature1=full_train_set1[my_id]
            feature2=full_train_set2[my_id]
        	target=full_train_set1[my_id]['target']
        else:
        	feature1=val_set1[my_id]
            feature2=val_set2[my_id]
        	target=val_set[my_id]['target']
        data=(feature1,feature2)
        return (data,target) 
       
    def __len__(self):
        return len(self.names)



def train(epoch,train_loader,optimizer,criterian):
    print('Training epoch..',epoch)
    model.train()
    torch.cuda.empty_cache()
    train_loss=0
    b_cnt=0
    for data,target in train_loader:
        b_cnt=b_cnt+1
        data=list(data)
        data[0]=data[0].cuda()
        data[1]=data[1].cuda()
        target=target.cuda().float()
        optimizer.zero_grad()
        pred=model(data).cuda()
        loss=criterian(pred,target).cuda()
        train_loss+=math.sqrt(loss.item()) 
        loss.backward()
        optimizer.step()
        torch.cuda.empty_cache()
        gc.collect()
    torch.cuda.empty_cache()
    return train_loss/float(b_cnt)



def run():
    torch.cuda.empty_cache()
    n_epochs=100
    print("Running...")
    global n_batches
    n_batches=100
    global model
    model=my_model().cuda()
    model=nn.DataParallel(model,device_ids=[0,1,2])
    criterian=nn.MSELoss().cuda()
    train_ids=find_dataset(full_train_set1) # train_ids is a list 
    l_rate=0.001
    optimizer=torch.optim.Adam(model.parameters(), lr=l_rate)
    train_loader = torch.utils.data.DataLoader(dataset=MyCustomDataset(train_ids,train=True),
                                                batch_size=n_batches,
                                                shuffle=True)

    for epoch in range(1,n_epochs+1):
        gc.collect()
        train_loss=train(epoch,train_loader,optimizer,criterian)
        print('Epoch :'+str(epoch)+': Train rmse:',train_loss)

run()
</code></pre>

username: justusschock
cooked: <p>Since the gradient update is usually done at CPU while Forward and backward are done on GPU, you could remove the <code>.cuda()</code> call at these parts of your code</p>
<aside class="quote no-group" data-username="Gkv" data-post="9" data-topic="24475">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/439d5e/48.png" class="avatar"> Gkv:</div>
<blockquote>
<p><code>loss=criterian(pred,target).cuda()</code></p>
</blockquote>
</aside>
<aside class="quote no-group" data-username="Gkv" data-post="9" data-topic="24475">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/439d5e/48.png" class="avatar"> Gkv:</div>
<blockquote>
<p><code>criterian=nn.MSELoss().cuda()</code></p>
</blockquote>
</aside>
<p>and put this part of your code before moving the model to GPU:</p>
<aside class="quote no-group" data-username="Gkv" data-post="9" data-topic="24475">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/439d5e/48.png" class="avatar"> Gkv:</div>
<blockquote>
<p><code>optimizer=torch.optim.Adam(model.parameters(), lr=l_rate)</code></p>
</blockquote>
</aside>
<p>Your <code>run</code>-function would now look like this:</p>
<pre><code class="lang-auto">def run():
    torch.cuda.empty_cache()
    n_epochs=100
    print("Running...")
    global n_batches
    n_batches=100
    global model
    model=my_model()
    optimizer=torch.optim.Adam(model.parameters(), lr=l_rate)

    model = model.cuda()

    model=nn.DataParallel(model,device_ids=[0,1,2])
    criterian=nn.MSELoss().cuda()
    train_ids=find_dataset(full_train_set1) # train_ids is a list 
    l_rate=0.001
    
    train_loader = torch.utils.data.DataLoader(dataset=MyCustomDataset(train_ids,train=True),
                                                batch_size=n_batches,
                                                shuffle=True)

    for epoch in range(1,n_epochs+1):
        gc.collect()
        train_loss=train(epoch,train_loader,optimizer,criterian)
        print('Epoch :'+str(epoch)+': Train rmse:',train_loss)

</code></pre>
<p>This would prevent loss function and optimizer from living on GPU (and thus decrease the GPU memory usage). However, since you run OOM on CPU, I would first try to load only the parts of your dataset you need just-in-time instead of loading the whole (probably huge) datasets before and cache them. Saying that, your CPU RAM should not increase but stay approx. at the same level of usage.</p>

username: Gkv
cooked: <p>Hi,<br>
I included those .cuda() after this problem occurred. Previously without this also the problem was there. My feature dictionaries are around 6gb in total and i am running this on a system with 251 gb RAM. Then how this will happen?. Is there any problem with the global variables?</p>

username: justusschock
cooked: <p>Can you try to run it without global variables (I.E. passing the model to the function as parameter and loading your data inside your Dataset’s <code>__init__</code>)? I usually try to avoid global variables.</p>

username: Gkv
cooked: <p>Passing the model to the function as a parameter is not solving the problem. The dataloader part I will do and update the results here.</p>

username: Gkv
cooked: <p>I tried loading the data inside <strong>init</strong>. But no change.</p>

username: Gkv
cooked: <p>Hi,<br>
The problem is in the Dataloader. When i simply iterate over the dataloader, the memory is increasing. I don’t know why? Any thoughts?<br>
Thanks</p>

username: zgahhblhc
cooked: <p>Solved my problem perfectly!</p>

username: imaginist
cooked: <p>Perfect! I have solved my question!</p>

username: nofreewill
cooked: <p>Thank you very much!!</p>

username: Rabeya_Akter
cooked: <p>I am facing the same issue, did you solve it?</p>

username: Axel-Jacobsen
cooked: <p>If you are using multiprocessing, see<br>
<a href="https://pytorch.org/docs/stable/data.html#multi-process-data-loading" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/data.html#multi-process-data-loading</a><br>
and</p><aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662" target="_blank" rel="noopener nofollow ugc">DataLoader num_workers &gt; 0 causes CPU memory from parent process to be replicated in all worker processes</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2018-10-29" data-time="13:23:59" data-timezone="UTC">01:23PM - 29 Oct 18 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/bfreskura" target="_blank" rel="noopener nofollow ugc">
          <img alt="bfreskura" src="https://avatars.githubusercontent.com/u/9371112?v=4" class="onebox-avatar-inline" width="20" height="20">
          bfreskura
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          high priority
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: dependency bug
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: multiprocessing
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: dataloader
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: molly-guard
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: memory usage
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          triaged
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container"># Editor note: There is a known workaround further down on this issue, which is <span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">to NOT use Python lists, but instead using something else, e.g., torch.tensor directly. See https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662 . You can use a numpy array, but it only fixes the issue for the `fork` start method. See https://github.com/pytorch/pytorch/issues/13246#issuecomment-1364587359 for more details


## 🐛 Bug

CPU memory will leak if the DataLoader `num_workers &gt; 0`.

## To Reproduce
Run the following snippet:

```
from torch.utils.data import Dataset, DataLoader
from PIL import Image
from torchvision import transforms
import os

class DataIter(Dataset):
    def __init__(self):
        path = "path/to/data"
        self.data = []

        for cls in os.listdir(path):
            for img in os.listdir(os.path.join(path, cls)):
                self.data.append(os.path.join(path, cls, img))

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        with Image.open(self.data[idx]) as img:
            img = img.convert('RGB')
            return transforms.functional.to_tensor(img)


train_data = DataIter()
train_loader = DataLoader(train_data, batch_size=300,
                          shuffle=True,
                          drop_last=True,
                          pin_memory=False,
                          num_workers=18)

for i, item in enumerate(train_loader):
    if i % 200 == 0:
        print(i)
```
## Expected behavior

CPU memory will gradually start increasing, eventually filling up the whole RAM. E.g., the process starts with around 15GB and fills up the whole 128GB available on the system. 
When the `num_workers=0`, RAM usage is constant.

## Environment

```
PyTorch version: 1.0.0.dev20181028
Is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 16.04.4 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
CMake version: version 3.5.1

Python version: 3.5
Is CUDA available: Yes
CUDA runtime version: 9.0.176
GPU models and configuration: 
GPU 0: GeForce GTX 1080 Ti
GPU 1: GeForce GTX 1080 Ti
GPU 2: GeForce GTX 1080 Ti

Nvidia driver version: 390.67
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.1.4

Versions of relevant libraries:
[pip] Could not collect
[conda] Could not collect

PIL.__version__
'5.3.0'
```

## Additional info
There are around 24 million images in the dataset and all image paths are loaded into a single list as presented in the above code snippet.

I have also tried multiple Pytorch (0.4.0 and 0.4.1) versions and the effect is the same.

cc @ezyang @gchanan @zou3519 @bdhirsh @jbschlosser @anjali411 @SsnL @VitalyFedyunin @ejguan</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>