Steady increase in CPU RAM memory

moth: <p>Bug report</p>
<ol>
<li>
<strong>Summary of problem:</strong> I’ve been encountering a steady increase in CPU RAM memory while using a PyTorch <code>DataLoader</code>. Only loading the data leads to an increase in CPU RAM which eventually crashes the notebook. Please note that I am actually using GPU and not CPU, hence my <strong>device</strong> is <code>cuda</code>. This notebook’s <code>torch</code> version is:</li>
</ol>
<ul>
<li>
<code>torch @ file:///tmp/torch/torch-1.13.0-cp37-cp37m-linux_x86_64.whl</code>.</li>
</ul>
<p>A very similar notebook which had not problem used <code>torch</code>:</p>
<ul>
<li><code>torch @ file:///tmp/torch/torch-1.11.0-cp37-cp37m-linux_x86_64.whl</code></li>
</ul>
<ol start="2">
<li>
<strong>Steps to reproduce:</strong><br>
Simple Dataset:</li>
</ol>
<pre><code class="lang-auto">from albumentations import Normalize
from albumentations.pytorch import ToTensorV2


class CustomDataset(Dataset):

    def __init__(self, dataframe, train, valid):

        self.dataframe = dataframe
        self.train = train
        self.valid = valid
        
        if self.train:
            self.transform = Compose([
                                      Normalize(p=1),
                                      ToTensorV2(p=1)])
        if self.valid:
            self.transform = Compose([
                                      Normalize(p=1),
                                      ToTensorV2(p=1)])

    def __len__(self):
        return len(self.dataframe)

    def __getitem__(self, index):
        idx = self.dataframe['image_id'][index] # Select image id
        image_path = self.dataframe['path'][index] # Get image path
        image = cv2.imread(image_path) # Read image
        image = self.transform(image=image) # Apply transforms
        image = image['image'] # Extract image from dictionary
        return (idx, image)
</code></pre>
<p>Loop:</p>
<pre><code class="lang-auto">for fold, (train_index, valid_index) in enumerate(kf.split(train_df)):
    train_data = train_df.iloc[train_index].reset_index(drop=True)
    train = CustomDataset(train_data, train=True, valid=False)
    train_loader = DataLoader(train,
                              batch_size=config.BATCH_SIZE_TRAIN,
                              shuffle=True,
                              num_workers=config.NUM_WORKERS,
                              drop_last=True)
    epochs = config.EPOCHS
    for epoch in range(epochs):
        for step, (idx, image) in enumerate(train_loader):
            image = torch.tensor(image, device=device, dtype=torch.float32)
</code></pre>
<p>After a couple hundred steps the notebook crashes and RAM is 100% used. It does not even complete one epoch.</p>

ptrblck: <p>I cannot reproduce the increase in host RAM using this code snippet:</p>
<pre><code class="lang-python">from albumentations import Normalize, Compose
from albumentations.pytorch import ToTensorV2
import cv2


class CustomDataset(Dataset):
    def __init__(self):
        self.transform = Compose([Normalize(p=1), ToTensorV2(p=1)])

    def __len__(self):
        return 100

    def __getitem__(self, index):
        image_path = "./image.jpeg"
        image = cv2.imread(image_path) # Read image
        image = self.transform(image=image) # Apply transforms
        image = image['image'] # Extract image from dictionary
        return (index, image)
    
dataset = CustomDataset()
loader = DataLoader(dataset, batch_size=10, num_workers=2, shuffle=True)

for epoch in range(1000):
    for step, (idx, image) in enumerate(loader):
        image = torch.tensor(image, device="cuda", dtype=torch.float32)
</code></pre>
<p>so I’m unsure if the issue is caused by the <code>dataframe</code> or another undefined object.<br>
Could you check if loading a single image as seen in my code also increases the RAM usage in your setup?</p>

moth: <p><a href="https://www.kaggle.com/code/alejopaullier/notebook26e53e7211?scriptVersionId=121570399" rel="noopener nofollow ugc">Here</a> is a minimal example on Kaggle to reproduce. I confirm that your snippet does not seem to cause this problem. However, I am unable to detect whether the root cause is the <code>dataframe</code> object or loading some particular images (can images be damaged (?)). Finally, I’ve noticed there is a discrepancy between the monitored RAM usage through the <code>psutil</code> library (via <code>psutil.virtual_memory().percent</code>) and the Kaggle monitor. On the first one, RAM seems constanst while in the second it keeps increasing.</p>