Out of CPU Memory

skyunyoo: <p>Hello,<br>
I’m currently experiencing a CPU Memory shortage, so I would like to get help.</p>
<p>I’m using about 400,000<em>64</em>64 (about 48G) and I have 32G GPU Memory.</p>
<p>If I train using the codes below, the memory usage is over 90%.<br>
The code of my custom dataset is below.</p>
<pre><code class="lang-auto">class trainDataset(torch.utils.data.Dataset):
    def __init__(self, i, data_path, augmentation=True):
        self.data_path = data_path
        self.data = np.load(data_path+'image{}.npy'.format(i)).astype(np.uint16)
        self.target = np.load(data_path+'label{}.npy'.format(i)).astype(np.uint8)
        self.augmentation = augmentation
    def __getitem__(self, index):
        x = self.data[index]
        y = self.target[index]
        x, y = self.transform(x, y)
            
        return x, y
    
    def transform(self, data, target):
        data, target = data_augmentation(data, target, self.augmentation)
        return data, target
    
    def __len__(self):
        return len(self.data)
</code></pre>
<p>I used ConcatDataset through other previous posts and concatenated into one dataset by creating 58 datasets as show below.</p>
<pre><code class="lang-auto">for i in range(58):
    i=i+1
    train_datasets.append(trainDataset(i,"D:/path/",augmentation=True))
traindataset = torch.utils.data.ConcatDataset(train_datasets)
print("Load TrainDataset Done")
</code></pre>
<p>And my training code is as below<br>
I also used “torch.set_num_threads()” to use Multi-Threads</p>
<pre><code class="lang-auto">def fit(epoch,model,data_loader,phase='train',volatile=False):
    torch.set_num_threads(2)
    if phase == 'train':
        model.train().to(device)
    if phase == 'valid':
        model.eval().to(device)
        
    running_loss = 0.0
    for batch_idx , (data,target) in enumerate(data_loader):
        inputs,target = data.to(device),target.to(device)
        optimizer.zero_grad()
        with torch.set_grad_enabled(phase == 'train'):         
            output = model(inputs).to(device)
            loss = criterion(output,target.long()).to(device)
            if phase == 'train':
                loss.backward()
                optimizer.step()
        running_loss += loss.data
    if phase == 'train':
        exp_lr_scheduler.step()   
    loss = running_loss/len(data_loader.dataset)
    print('{} Loss: {:.4f}'.format(
                phase, loss))
    return loss
</code></pre>
<pre><code class="lang-auto">init_state = copy.deepcopy(model.state_dict())
init_state_opt = copy.deepcopy(optimizer.state_dict())
init_state_lr = copy.deepcopy(exp_lr_scheduler.state_dict())

since = time.time()
    
train_losses = []
val_losses = []

print()
print('train : {}, valid : {}'.format(len(trainloader.dataset), len(validloader.dataset)))
early_stopping = EarlyStopping(patience=5, verbose=1)
for epoch in range(num_epochs):
    print()
    print('Epoch {}/{}'.format(epoch, num_epochs - 1))
    print('-' * 10)
    epoch_loss = fit(epoch,model,trainloader,phase='train')
    val_epoch_loss = fit(epoch,model,validloader,phase='valid')
    train_losses.append(epoch_loss)
    val_losses.append(val_epoch_loss)
        
    if early_stopping.validate(val_epoch_loss):
        break
</code></pre>
<p>I wonder if there is insufficient memory because of the size of the files. Or are there other ways to reduce memory?</p>
<p>Any help would be appreciated<br>
Thank!</p>

ptrblck: <p>How large is each <code>.npy</code> file and how many are you loading at once?</p>

skyunyoo: <p>Thank you for the reply!</p>
<p>Each .npy of input is less than 1GB and averages about 700MB.<br>
The label has a smaller capacity.<br>
The total train capacity combined with input and label is about 48GB.</p>
<p>There are a total of 58 input npy and label npy,<br>
For each dataset, one input and one label are loaded.<br>
A total of 58 datasets are created, and ConcatDataset is used to create one dataset (Final train dataset).</p>

ptrblck: <p>Thanks for the information!<br>
Since you are loading each <code>.npy</code> in the <code>Dataset.__init__</code> method, <code>ConcatDataset</code> will append all already initialized datasets and will thus use a huge amount of memory.</p>
<p>Would it be possible to create a loop over each '.npy<code>file and create a single</code>trainDataset<code>inside this loop for the current</code>.npy`?<br>
This would hold a single dataset in memory at any time and should reduce the memory foot print significantly.</p>

skyunyoo: <p>Thanks for the idea!</p>
<p>The idea you mentioned is not using ConcatDataset, but creating a single traindataset for each npy.<br>
After that, train 58 traindatasets in the train loop as one epoch.<br>
Is what I understand correct?</p>
<p>Thanks again for your help <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>

ptrblck: <p>Yes, you won’t be able to shuffle the samples between the different datasets, but it might be OK, if you shuffle inside each dataset.</p>

skyunyoo: <p>Thanks to this, my research area has expanded!<br>
Thank you very much <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=9" title=":+1:" class="emoji" alt=":+1:"> <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=9" title=":+1:" class="emoji" alt=":+1:"> <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=9" title=":+1:" class="emoji" alt=":+1:"></p>