Weird profiler stack trace : Not sure if GPU is enabled on a local machine

Monkey.Py: <p>Hi,<br>
I am currently working on profiling, learning about torch profiler and tensorboard using it. I am using this tutorial : <a href="https://pytorch.org/tutorials/intermediate/tensorboard_profiler_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">PyTorch Profiler With TensorBoard — PyTorch Tutorials 2.2.0+cu121 documentation</a>.</p>
<p>The thing is that I tried it using google colab &amp; my own local computer that has a RTX2080. When I run the exact tutorial code with colab I am obtaining a similar report, telling me about GPU stuff…<br>
Yet when I run on my machine (after a call of .cuda() I checked that my models &amp; data were on cuda and according to pytorch they are) I am obtaining the following result :<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/e/fe360ad5e146f635b47fe32bd7f3a8d1faec8f69.png" data-download-href="https://discuss.pytorch.org/uploads/default/fe360ad5e146f635b47fe32bd7f3a8d1faec8f69" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/fe360ad5e146f635b47fe32bd7f3a8d1faec8f69_2_690x272.png" alt="image" data-base62-sha1="AgR3SNJfWj590giWLvLrpYCTtxf" width="690" height="272" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/fe360ad5e146f635b47fe32bd7f3a8d1faec8f69_2_690x272.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/fe360ad5e146f635b47fe32bd7f3a8d1faec8f69_2_1035x408.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/e/fe360ad5e146f635b47fe32bd7f3a8d1faec8f69_2_1380x544.png 2x" data-dominant-color="FAEAE0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2417×954 113 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This seems to indicate that all the operation performed were performed on CPU (and the device is CPU), while my device is cuda.</p>
<p>I believe that it might be because I think I did not install cudatoolkit on my machine (but I did install torch using pip), but can this line :</p>
<pre><code class="lang-auto">device = cuda if torch.cuda.is_available() else cpu
</code></pre>
<p>return cuda if cudatoolkit is not installed on my machine ? Is it possible to put models &amp; data on cuda and still perform operation on cpu ?</p>
<p>And if it is and if I am performing operation on cpu : I noticed that it is still way quicker to have model &amp; data on cuda than on cpu, how can it be quicker ?</p>
<p>Thank you for your time and help</p>

ptrblck: <aside class="quote no-group" data-username="Monkey.Py" data-post="1" data-topic="191021">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/monkey.py/48/65032_2.png" class="avatar"> Monkey.Py:</div>
<blockquote>
<p>return cuda if cudatoolkit is not installed on my machine ?</p>
</blockquote>
</aside>
<p>Yes, since PyTorch binaries ship with their own CUDA dependencies and you thus won’t need to install a local CUDA toolkit.</p>
<aside class="quote no-group" data-username="Monkey.Py" data-post="1" data-topic="191021">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/monkey.py/48/65032_2.png" class="avatar"> Monkey.Py:</div>
<blockquote>
<p>Is it possible to put models &amp; data on cuda and still perform operation on cpu ?</p>
</blockquote>
</aside>
<p>If the model and data were moved to the GPU their operations will be performed on the GPU. Other tensors (on the CPU) can still perform operations on the CPU.</p>
<aside class="quote no-group" data-username="Monkey.Py" data-post="1" data-topic="191021">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/monkey.py/48/65032_2.png" class="avatar"> Monkey.Py:</div>
<blockquote>
<p>And if it is and if I am performing operation on cpu : I noticed that it is still way quicker to have model &amp; data on cuda than on cpu, how can it be quicker ?</p>
</blockquote>
</aside>
<p>If you moved the model and data to the GPU all operations will be performed on the GPU.</p>
<p>To your profiler question: I’m not familiar enough with the built-in profiler and use Nsight Systems instead, so you might also give it a try.</p>

Monkey.Py: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="191021">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Nsight Systems</p>
</blockquote>
</aside>
<p>Thank you, I will have a look and make a profiling using it, we’ll see if this one understand that operations are done on GPU.</p>

Osama_Abu_Hamdan: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="191021">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Yes, since PyTorch binaries ship with their own CUDA dependencies and you thus won’t need to install a local CUDA toolkit.</p>
</blockquote>
</aside>
<p>This answer helped me to get GPU recognized by the profiler with a side effect.</p>
<p>This command gets the “Distributed” view to appear, but the GPU is not recognized.</p>
<pre data-code-wrap="bash"><code class="lang-bash">srun nsys profile --gpu-metrics-device=0 --sample=none --cpuctxsw=none --force-overwrite true --nic-metrics=true --trace=cuda,nvtx,cudnn --output=logs/${EXP_NAME}/nsys_logs_%h torchrun \
--nnodes $NUM_NODES \
--nproc_per_node 1 \
--rdzv_id $RANDOM \
--rdzv_backend c10d \
--rdzv_endpoint $MASTER_ADDR:$MASTER_PORT \
cifar10_with_resnset.py --epochs=10
</code></pre>
<p>While this command gets the “Distributed” view to diappear, and the GPU to be recognized.</p>
<pre data-code-wrap="bash"><code class="lang-bash">torchrun \
--nnodes $NUM_NODES \
--nproc_per_node 1 \
--rdzv_id $RANDOM \
--rdzv_backend c10d \
--rdzv_endpoint $MASTER_ADDR:$MASTER_PORT \
cifar10_with_resnset.py --epochs=10
</code></pre>
<p>My PyTorch Code</p>
<pre><code class="lang-auto">import argparse

import torch
import torch.distributed as dist
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
from torch.cuda.amp import GradScaler, autocast
from torch.nn.parallel import DistributedDataParallel
from torch.utils import data
from torchvision import models

precisions = {
    "fp16": torch.float16,
    "fp32": torch.float32,
}


def train(train_sampler, epoch, device, train_loader, model, loss_criterion, optimizer, profiler, scaler, precision):
    train_sampler.set_epoch(epoch)
    model.train()
    for batch_idx, (inputs, targets) in enumerate(train_loader):
        profiler.step()
        inputs, targets = inputs.cuda(device), targets.cuda(device)
        optimizer.zero_grad()

        with autocast(dtype=precision):
            outputs = model(inputs)
            loss = loss_criterion(outputs, targets)

        scaler.scale(loss).backward()
        scaler.step(optimizer)
        scaler.update()

        if batch_idx % 10 == 0:
            current_samples = batch_idx * len(inputs)
            total_samples = len(train_sampler)
            progress = 100.0 * current_samples / total_samples
            print(f'Epoch: {epoch} [{current_samples}/{total_samples} ({progress:.0f}%)]\tLoss: {loss.item():.6f}')


def validate(epoch, device, validation_loader, model, loss_criterion):
    model.eval()
    test_loss = 0
    correct = 0
    with torch.no_grad():
        for inputs, targets in validation_loader:
            inputs, targets = inputs.cuda(device), targets.cuda(device)
            outputs = model(inputs)
            test_loss += loss_criterion(outputs, targets).item()
            _, predicted = outputs.max(1)
            correct += predicted.eq(targets).sum().item()

    test_loss /= len(validation_loader.dataset)
    accuracy = 100. * correct / len(validation_loader.dataset)
    print(f'\nValidation. Epoch: {epoch}, Loss: {test_loss:.4f}, Accuracy: ({accuracy:.2f}%)\n')


def cleanup():
    dist.destroy_process_group()


def get_data(rank, world_size, args):
    transform = transforms.Compose([
        transforms.RandomHorizontalFlip(),
        transforms.RandomCrop(32, padding=4),
        transforms.ToTensor(),
        transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
    ])

    train_set = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
    train_sampler = data.distributed.DistributedSampler(train_set, num_replicas=world_size, rank=rank)
    train_loader = data.DataLoader(train_set, sampler=train_sampler, batch_size=args.batch_size, num_workers=1,
                                   persistent_workers=True)

    validation_set = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform)
    validation_loader = data.DataLoader(validation_set, batch_size=args.batch_size, num_workers=1,
                                        persistent_workers=True)

    return train_sampler, train_loader, validation_loader


def main(args, rank, world_size):
    torch.manual_seed(0)
    train_sampler, train_loader, validation_loader = get_data(rank, world_size, args)

    device = torch.device("cuda:0")
    print(f"Rank {rank}: Using CUDA device {device}")

    model = models.resnet18(weights=None, num_classes=10)
    model.conv1 = nn.Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
    model = model.cuda(device)

    ddp_model = DistributedDataParallel(model)
    scaler = GradScaler()

    loss_criterion = nn.CrossEntropyLoss().cuda(device)
    optimizer = torch.optim.AdamW(model.parameters(), lr=0.001, weight_decay=5e-4)
    scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=30, gamma=0.1)

    print(torch.distributed.is_nccl_available())
    print(torch.distributed.get_backend_config())

    with torch.profiler.profile(
            activities=[torch.profiler.ProfilerActivity.CPU, torch.profiler.ProfilerActivity.CUDA],
            schedule=torch.profiler.schedule(wait=1, warmup=1, active=10, repeat=1),
            on_trace_ready=torch.profiler.tensorboard_trace_handler(f'./logs/{args.exp_name}'),
            record_shapes=True,
            profile_memory=True,
            with_stack=True,
    ) as profiler:
        for epoch in range(args.epochs):
            train(train_sampler, epoch, device, train_loader, ddp_model, loss_criterion, optimizer, profiler, scaler,
                  precisions[args.precision])
            validate(epoch, device, validation_loader, ddp_model, loss_criterion)
            scheduler.step()
        cleanup()


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--epochs", type=int, default=10)
    parser.add_argument("--batch-size", type=int, default=512)
    parser.add_argument("--model", type=str, default="resnet18")
    parser.add_argument("--precision", type=str, default="fp32")
    parser.add_argument("--exp-name", type=str, default="test")
    args = parser.parse_args()

    # distributed
    torch.distributed.init_process_group(backend='nccl', init_method='env://')
    rank = torch.distributed.get_rank()
    world_size = torch.distributed.get_world_size()

    main(args, rank, world_size)

</code></pre>

ptrblck: <p>Check why your slurm environment seems to have issues with your GPU or use Nsight Systems directly on <code>torchrun</code>.</p>

Osama_Abu_Hamdan: <aside class="quote no-group" data-username="ptrblck" data-post="5" data-topic="191021">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>issues with your GPU</p>
</blockquote>
</aside>
<p>Like what issues?</p>
<aside class="quote no-group" data-username="ptrblck" data-post="5" data-topic="191021">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Nsight Systems directly on <code>torchrun</code></p>
</blockquote>
</aside>
<p>How?</p>
<p>Thanks!</p>