DataLoader increases RAM usage every iteration

mlagunas: <p>Hello,<br>
I’m running into troubles while training a U-net like model. I defined my own dataset class as follows:</p>
<pre><code>class CARVANA(Dataset):
    """
        CARVANA dataset that contains car images as .jpg. Each car has 16 images
        taken in different angles and a unique id: id_01.jpg, id_02.jpg, ..., id_16.jpg
        The labels are provided as a .gif image that contains the manually cutout mask
        for each training image
    """

    def __init__(self, root, train=True, transform=None):
        """

        :param root: it has to be a path to the folder that contains the dataset folders
        :param train: boolean true if you want the train set false for the test one
        :param transform: transform the images and labels
        """

        # initialize variables
        self.root = os.path.expanduser(root)
        self.transform = transform
        self.train = train
        self.data, self.labels = [], []

        def load_images(path, data):
            """
            loads all the images in path and stores them in data.

            :param path:
            :param data:
            :return: tensor with all the images from path loaded
            """

            # read path content
            images_dir = [f for f in os.listdir(path) if isfile(join(path, f))]
            images_dir.sort()

            # load images
            for image in tqdm(images_dir, desc="loading data"):
                    data.append(Image.open(join(path, image)))

            return data

        if self.train:
            self.data = load_images(self.root + "/train", self.data)
            self.labels = load_images(self.root + "/train_masks", self.labels)
        else:
            self.data = load_images(self.root + "/test", self.data)
            self.labels = None

    def __getitem__(self, index):
        """

        :param index:
        :return: tuple (img, target) with the input data and its label
        """

        # load image and labels
        img = self.data[index]
        target = self.labels[index] if self.train else None

        # apply transforms to both
        if self.transform is not None:
            img = self.transform(img)
            target = self.transform(target)

        return img, target

    def __len__(self):
        return len(self.data)
</code></pre>
<p>Then I load it using:</p>
<pre><code>train_dataset = dsets.CARVANA(root="./data/",
                              train=True,
                              transform=transforms.Compose([
                                  transforms.Scale(256),
                                  transforms.RandomCrop(256),
                                  transforms.ToTensor(),
                              ])
                              )
train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
                                           batch_size=batch_size,
                                           shuffle=True,
                                           pin_memory=False,
                                           num_workers=1)
</code></pre>
<p>And finally, I run a simple training loop</p>
<pre><code>for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        # Convert torch tensor to Variable
        images = Variable(images.cuda())
        labels = Variable(labels.cuda())

        # Forward + Backward + Optimize
        optimizer.zero_grad()  # zero the gradient buffer
        outputs = unet(images)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        gc.collect()
</code></pre>
<p>I have tracked which lines increase the RAM usage. In particular, only in <code>for i, (images, labels) in enumerate(train_loader):</code> the usage increases about 200MB each iteration making it really slow after a few (I just have 32GB). I tried to make both <code>images, labes = None, None</code> before <code>gc.collect()</code> but it did not help.</p>
<p>Any ideas on what the problem could be?</p>
<p>Probably it is a stupid mistake I am doing while loading the batch in the loop since I am pretty new to pytorch, but I could not find any similar error in the forum (error looks similar to <a href="https://discuss.pytorch.org/t/data-loaders-memory-issues-and-circular-references/6530/5">Data loaders, memory issues and circular references</a>).</p>
<p>Thanks in advance!<br>
Manu.</p>

trypag: <p>Is there any reason why you load all of your images during <code>init</code> ?<br>
Depending on the number of images, it can explain why the execution requires so much memory. You should rather provide an index based on the image address, and load the image while calling <code>_getitem_</code>. The purpose of using a <code>Dataset</code> is to load the data on demand, thus not requiring all the images to be loaded in memory (that’s what you are doing).<br>
Also you can make sure only one copy of your tensor is shared across workers by using <code>.share_memory_()</code> on the underlying tensors, if you have. For example, if you had created a tensor of positions during init, then you would want that tensor to be shared while the <code>Dataloader</code>'s workers create the batches.</p>

mlagunas: <blockquote>
<p>Is there any reason why you load all of your images during init ?</p>
</blockquote>
<p>That was the error (and it makes lots of sense <img src="https://discuss.pytorch.org/images/emoji/apple/sweat.png?v=12" title=":sweat:" class="emoji" alt=":sweat:" loading="lazy" width="20" height="20">) .<br>
Now, I store the image paths during <code>__init__</code> and load the images during <code>__getitem__</code>, as you mentioned. It works perfectly.</p>
<p>Thanks,</p>

longyangqi: <p>I wanna ask a question…<br>
When I running my code, I find the cached mem increased every iteration, until the RAM is full.<br>
Is that a common issue or a memory leak?<br>
when i use “top” command in terminal, I see that:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/0d7ad57c06bdd577cef0ca1c6140f9e74dfe567a.png" data-download-href="https://discuss.pytorch.org/uploads/default/0d7ad57c06bdd577cef0ca1c6140f9e74dfe567a" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/0/0d7ad57c06bdd577cef0ca1c6140f9e74dfe567a.png" alt="image" data-base62-sha1="1VfnpqLZlcCdo8GIo78cfTsg80W" width="690" height="125" data-dominant-color="242323"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1053×192 11.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks!</p>

gebbissimo: <p><a class="mention" href="/u/longyangqi">@longyangqi</a>  Maybe using tqdm is problematic? See <a href="https://github.com/tqdm/tqdm/issues/746" rel="nofollow noopener">https://github.com/tqdm/tqdm/issues/746</a></p>