Questions about GPU allocate and cache

skyunyoo: <p>Hello.<br>
I have a question about GPU allocate and cache.</p>
<p>I have seen GPU allocation and cache using code like below (Below is nothing loaded on the GPU)</p>
<pre><code class="lang-auto">device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print('Using device:', device)
print()

if device.type == 'cuda':
    print(torch.cuda.get_device_name(0))
    print('Memory Usage:')
    print('Allocated:', round(torch.cuda.memory_allocated(0)/1024**3,1), 'GB')
    print('Cached:   ', round(torch.cuda.memory_cached(0)/1024**3,1), 'GB')
</code></pre>
<pre><code class="lang-auto">Using device: cuda

TITAN RTX
Memory Usage:
Allocated: 0.0 GB
Cached:    0.0 GB
</code></pre>
<p>However, when I loaded the model and inputs(targets) into the GPU and checked the code above for each step, I found something strange.</p>
<pre><code class="lang-auto">model = model.to(device)

for batch_idx , (data,target) in enumerate(data_loader):
        inputs,target = data.to(device),target.to(device)
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/8/98a46504f6eb968567599517720abcc61cffc3db.png" alt="memory use" data-base62-sha1="lMkMWbhW6YalyOO48ZJIN3mZHbJ" width="148" height="319"></p>
<p>As shown in the picture, during training and validation, only GPU memory was allocated 0.1GB and most of them went into the cache.</p>
<p>Because of this, CPU usage seems to increase abnormally.</p>
<p>I loaded both the model and inputs(targets) into the GPU. Why is the GPU allocation only 0.1GB?<br>
How do I increase GPU allocation?</p>

albanD: <p>Hi,</p>
<p>If your model is relatively small, it is not abnormal for it only to use 100MB.<br>
What takes a lot of memory are all the intermdiate results needed for gradient computation. You can check the max allocated memory to see the peak usage.</p>

skyunyoo: <p>Thanks for reply!<br>
As you said, I saw peak usage and use about 8GB.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/8/78f074987b8b5cbf61a06bbe03ced0d8e86a9dae.png" alt="캡처" data-base62-sha1="hfSt8ek1xUUlXb0o6wFMRNUsHSK" width="157" height="418"></p>
<p>But another question arises: the GPU sees it used, but the CPU usage is weird.</p>
<p>Until I run the model, the CPU usage is 30-40%, but if I run the model with the following code, the CPU usage is close to 90-100%.</p>
<pre><code class="lang-auto">model = UNet(n_class=2)

if torch.cuda.is_available():
    model = model.cuda()

class_weights = torch.tensor([1.0, 2.0]).cuda()
criterion = nn.CrossEntropyLoss(weight=class_weights)
optimizer = optim.Adam(model.parameters(),lr=0.0005)
exp_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=10, gamma=0.1)
</code></pre>
<pre><code class="lang-auto">def fit(epoch,model,data_loader,phase='train',volatile=False):
    if phase == 'train':
        model.train()
    if phase == 'valid':
        model.eval()
        
    running_loss = 0.0
    for batch_idx , (data,target) in enumerate(data_loader):
        if is_cuda:
            inputs,target = data.cuda(),target.cuda()
        if phase == 'train':
            optimizer.zero_grad()       
        output = model(inputs)
        loss = criterion(output,target.long())           
        running_loss += loss.data.item()
        
        if phase == 'train':
            loss.backward()
            optimizer.step()
            exp_lr_scheduler.step()
    
    loss = running_loss/len(data_loader.dataset)

    print('{} Loss: {:.4f}'.format(
                phase, loss))
    return loss
</code></pre>
<pre><code class="lang-auto">init_state = copy.deepcopy(model.state_dict())
init_state_opt = copy.deepcopy(optimizer.state_dict())
init_state_lr = copy.deepcopy(exp_lr_scheduler.state_dict())

since = time.time()
    
train_losses = []
val_losses = []

print()
print('train : {}, valid : {}'.format(len(trainloader.dataset), len(validloader.dataset)))
early_stopping = EarlyStopping(patience=5, verbose=1)
for epoch in range(num_epochs):
    print()
    print('Epoch {}/{}'.format(epoch, num_epochs - 1))
    print('-' * 10)
    epoch_loss = fit(epoch,model,trainloader,phase='train')
    val_epoch_loss = fit(epoch,model,validloader,phase='valid')
    train_losses.append(epoch_loss)
    val_losses.append(val_epoch_loss)
        
    if early_stopping.validate(val_epoch_loss):
        break
torch.save(model.state_dict(),'U-Net_Result.pt')
print()      
result_graph()
    
time_elapsed = time.time() - since
print()
print('Training complete in {:.0f}m {:.0f}s'.format(
time_elapsed // 60, time_elapsed % 60))
</code></pre>
<p>At first I was suspicious of data augmentation and the size of the data, so I reduced the size without applying data augmentation, but still CPU usage is close to 90-100%.</p>
<p>Can you give me some advice on CPU usage?</p>

albanD: <p>CPU usage of 100% (that means a single core used), is expected. The CPU has the task of running the python code + queue up work for the GPU. If the model is small, this is expected that one thread is fully used on the CPU.</p>

skyunyoo: <p>Thank you for your help.<br>
Thanks to you, I have been studying deeper.<br>
I used your <code>torch.set_num_threads (2)</code>’ as a reference to your previous article, and now I have significantly reduced cpu usage.</p>
<p>Again Thanks! <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"></p>