Memory usage keeps increasing in Dataloader (in very simple code)

zhiweifang: <p>Hi,</p>
<p>I create a dataloader to load features from local files by their file pathes.<br>
The dataloader can be simplfied as:</p>
<p>(P.S. All codes were tested on Pytorch 1.0.0 and Pytorch 1.0.1. Memory capacity of my machine is <code>256Gb</code>)</p>
<pre><code class="lang-python">import numpy as np
import torch
import torch.utils.data as data
import time

class MyDataSet(data.Dataset):

  def __init__(self):

    super(MyDataSet, self).__init__()

    # Assume that the self.infoset here contains the description information about the dataset.
    # Here it is a list of strings. I set it aoubt 8Gb in memory.
    # In my real project, this infoset may be 40Gb in memory.
    self.infoset = [str(i).zfill(1024) for i in range(len(self))]


  def __getitem__(self, index):

    info = self.infoset[index]  # problem is here

    items = {}
    items['features'] = self.load_feature(info)

    return items

  def load_feature(self, info):
    '''
    Load feature from files
    '''
    feature = torch.Tensor(np.ones([8, 4, 2], dtype=np.float32))

    return feature

  def __len__(self):

    return 8000000

dataset = MyDataSet()

dataloader = data.DataLoader(dataset, batch_size=1024, shuffle=True, num_workers=16, pin_memory=True)

while True:

  for i, sample in enumerate(dataloader):

    print(i, len(dataloader))

    time.sleep(0.05) # slow down the process to see the mem-usage increasing during one epoch

</code></pre>
<p>During each epoch, the memory usage is about <code>13GB</code> at the very beginning and keeps inscreasing and finally up to about <code>46Gb</code>. Although it will decrease to <code>13GB</code> at the beginning of next epoch, this problem is serious because in my real project the infoset is about <code>40Gb</code> due to the large number of samples and finally leads to <code>Out of Memory</code> (OOM).</p>
<p>I have found that the problem is caused by the first line of <code>MyDataset.__getitem__()</code>, in the following code, infoset is just read but not used, but the same problem still happen.</p>
<pre><code class="lang-python">class MyDataSet(data.Dataset):

  def __init__(self):

    super(MyDataSet, self).__init__()

    # Assume that the self.infoset here contains the description information about the dataset.
    # Here it is a list of strings. I set it aoubt 8Gb in memory.
    # In my real project, this infoset may be 40Gb in memory.
    self.infoset = [str(i).zfill(1024) for i in range(len(self))]


  def __getitem__(self, index):

    info = self.infoset[index]  # problem is here

    items = {}
    # items['features'] = self.load_feature(info)

    return items

  def load_feature(self, info):
    '''
    Load feature from files
    '''
    feature = torch.Tensor(np.ones([8, 4, 2], dtype=np.float32))

    return feature

  def __len__(self):

    return 8000000

dataset = MyDataSet()

dataloader = data.DataLoader(dataset, batch_size=1024, shuffle=True, num_workers=16, pin_memory=True)

while True:

  for i, sample in enumerate(dataloader):

    print(i, len(dataloader))

    time.sleep(0.05) # slow down the process to see the mem-usage increasing during one epoch

</code></pre>
<p>As a contrast, when I comment the first line of <code>MyDataset.__getitem__()</code>, the memory usage keeps stable:</p>
<pre><code class="lang-python">class MyDataSet(data.Dataset):

  def __init__(self):

    super(MyDataSet, self).__init__()

    # Assume that the self.infoset here contains the description information about the dataset.
    # Here it is a list of strings. I set it aoubt 8Gb in memory.
    # In my real project, this infoset may be 40Gb in memory.
    self.infoset = [str(i).zfill(1024) for i in range(len(self))]


  def __getitem__(self, index):

    # info = self.infoset[index]  # problem is here
    info = 'fake info'

    items = {}
    items['features'] = self.load_feature(info)

    return items

  def load_feature(self, info):
    '''
    Load feature from files
    '''
    feature = torch.Tensor(np.ones([8, 4, 2], dtype=np.float32))

    return feature

  def __len__(self):

    return 8000000

dataset = MyDataSet()

dataloader = data.DataLoader(dataset, batch_size=1024, shuffle=True, num_workers=16, pin_memory=True)

while True:

  for i, sample in enumerate(dataloader):

    print(i, len(dataloader))

    time.sleep(0.05) # slow down the process to see the mem-usage increasing during one epoch
</code></pre>
<p>Any reasons and suggestions about this problems?</p>

sqrt: <p>Did you manage to find a solution? I’m hitting the same issue.</p>