PyTorch profiler with Tensorboard not capturing Dataloader time

Deepanker_Gupta1: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/6/466459c861755091c2e20e51ae3331f1d0629790.png" data-download-href="https://discuss.pytorch.org/uploads/default/466459c861755091c2e20e51ae3331f1d0629790" title="Screenshot 2023-01-10 at 12.56.58 AM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/466459c861755091c2e20e51ae3331f1d0629790_2_690x353.png" alt="Screenshot 2023-01-10 at 12.56.58 AM" data-base62-sha1="a2IqfkMmuvGEUq7gibRXfTjX1V6" width="690" height="353" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/466459c861755091c2e20e51ae3331f1d0629790_2_690x353.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/466459c861755091c2e20e51ae3331f1d0629790_2_1035x529.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/6/466459c861755091c2e20e51ae3331f1d0629790_2_1380x706.png 2x" data-dominant-color="E1DDE8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-01-10 at 12.56.58 AM</span><span class="informations">2990×1530 367 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Issue → PyTorch profiler not capturing Dataloader time and runtime. Always shows 0.<br>
Code used → I have used the code given in official PyTorch profiler documentation ( <a href="https://pytorch.org/tutorials/intermediate/tensorboard_profiler_tutorial.html" rel="noopener nofollow ugc">PyTorch documentation</a>)</p>
<p>Hardware Used-&gt; Nvidia AI100 gpu<br>
PyTorch version-&gt; 1.13.0+cu117<br>
PyTorch tensorboard profiler version → 0.4.1</p>

Deepanker_Gupta1: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> can you please help me out here.</p>

ptrblck: <p>I’m not familiar enough with the Kineto profiler and don’t know why it’s not showing the <code>DataLoader</code> workload. As an alternative, you could use <code>nvtx</code> ranges and profile your workload with Nsight Systems as described in <a href="https://dev-discuss.pytorch.org/t/using-nsight-systems-to-profile-gpu-workload/59">this post</a>.</p>

Deepanker_Gupta1: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/b/1b4662d60285a3f1c6e724349bb5eed606bfc4d9.png" data-download-href="https://discuss.pytorch.org/uploads/default/1b4662d60285a3f1c6e724349bb5eed606bfc4d9" title="Screenshot 2023-01-11 at 3.17.18 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b4662d60285a3f1c6e724349bb5eed606bfc4d9_2_690x402.png" alt="Screenshot 2023-01-11 at 3.17.18 PM" data-base62-sha1="3ThGYzV6Y4hWssX1CAg14xLVJj3" width="690" height="402" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b4662d60285a3f1c6e724349bb5eed606bfc4d9_2_690x402.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b4662d60285a3f1c6e724349bb5eed606bfc4d9_2_1035x603.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/b/1b4662d60285a3f1c6e724349bb5eed606bfc4d9_2_1380x804.png 2x" data-dominant-color="DEDFDF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-01-11 at 3.17.18 PM</span><span class="informations">2124×1238 413 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , thanks for telling the alternative, I tried the nsys command and generated the output as well which I opened in Nsight systems but got nvtx and cuda errors.</p>

ptrblck: <p>Did you follow my tutorial and were you able to profile the example code using the provided commands?</p>

Deepanker_Gupta1: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/f/7fbe0c7313d0134fea47f69e68e483cdf0605ee1.png" data-download-href="https://discuss.pytorch.org/uploads/default/7fbe0c7313d0134fea47f69e68e483cdf0605ee1" title="Screenshot 2023-01-11 at 4.10.10 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7fbe0c7313d0134fea47f69e68e483cdf0605ee1_2_690x448.png" alt="Screenshot 2023-01-11 at 4.10.10 PM" data-base62-sha1="ie3OAWaED4Z8mjaAqRhj9cZFks1" width="690" height="448" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7fbe0c7313d0134fea47f69e68e483cdf0605ee1_2_690x448.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7fbe0c7313d0134fea47f69e68e483cdf0605ee1_2_1035x672.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7fbe0c7313d0134fea47f69e68e483cdf0605ee1_2_1380x896.png 2x" data-dominant-color="DEDEDE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-01-11 at 4.10.10 PM</span><span class="informations">2076×1348 402 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Yes <a class="mention" href="/u/ptrblck">@ptrblck</a> ,<br>
I needed to modify the command a little bit as your command<br>
<code>nsys profile -w true -t cuda,nvtx,osrt,cudnn,cublas -s cpu  --capture-range=cudaProfilerApi --stop-on-range-end=true --cudabacktrace=true -x true -o my_profile python main.py</code><br>
was giving the following error:-<br>
unrecognised option ‘–stop-on-range-end=true’</p>
<p>so I changed it to<br>
<code>nsys profile -w true -t cuda,nvtx,osrt,cudnn,cublas -s cpu  --capture-range=cudaProfilerApi --capture-range-end=stop-shutdown  --cudabacktrace=true -x true -o my_profile python main.py</code><br>
I am getting  these warnings like Not all NVTX events might have been collected etc. with the example you have shared.</p>

Deepanker_Gupta1: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> it worked now, thanks once again, one last qsn can we add custom labels to parallel/async tasks as well? like when  num_workers=2 in data loader<br>
End goal is to figure out visually if a particular task is happening synchronously or asynchronously.</p>

ptrblck: <p>I would assume you could add nvtx ranges inside the <code>Dataset.__getitem__</code> and use the worker id for the range tag. This should show up in the timeline for each worker of the <code>DataLoader</code>.<br>
I haven’t tried this out yet, so let me know if it works.</p>

Deepanker_Gupta1: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> it worked, but the amount of data getting loaded isn’t changing when I change the prefetch factor keeping num of workers as 3.<br>
for example in the below image 128 get items were called by each worker, and 64 additional by the first worker as there were only 128*3+64 images,shouldn’t  with prefetch factor 2 the 64 get items call under batch 0 happen during data loading<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/5/9581646219fa490d7424f93604f7d55c300eaeb2.png" data-download-href="https://discuss.pytorch.org/uploads/default/9581646219fa490d7424f93604f7d55c300eaeb2" title="Screenshot 2023-01-18 at 1.59.21 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/5/9581646219fa490d7424f93604f7d55c300eaeb2_2_690x190.png" alt="Screenshot 2023-01-18 at 1.59.21 PM" data-base62-sha1="lkAmAU9wKqKbsXxDCo6szQkjiPE" width="690" height="190" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/5/9581646219fa490d7424f93604f7d55c300eaeb2_2_690x190.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/5/9581646219fa490d7424f93604f7d55c300eaeb2_2_1035x285.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/5/9581646219fa490d7424f93604f7d55c300eaeb2_2_1380x380.png 2x" data-dominant-color="EBECED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-01-18 at 1.59.21 PM</span><span class="informations">2806×776 127 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Maxime_G: <p>I have the same symptoms :</p>
<ul>
<li>Dataloader time not reported/catpured</li>
<li>I catpure 20 iteration, however tensorboard only shows 1 big and long iteration, (with all the time added up )</li>
</ul>
<p>ENV:</p>
<ul>
<li>pytorch 1.13.1</li>
<li>Nvidia Titan, ubuntu 20.04, tensorboard 2.11.2</li>
</ul>

Deepanker_Gupta1: <p><a class="mention" href="/u/maxime_g">@Maxime_G</a> please use this method <a href="https://dev-discuss.pytorch.org/t/using-nsight-systems-to-profile-gpu-workload/59" rel="noopener nofollow ugc">https://dev-discuss.pytorch.org/t/using-nsight-systems-to-profile-gpu-workload/59</a></p>
<p>if you are starting the training in a separate thread make sure you use the “spawn” as multiprocessing startup method.</p>
<p>Tensorboard didn’t work for me, there is a bug in the output file generated using PyTorch profiler</p>

Xinchengzelin: <p>Could you give me some advices to show the ''DATA LOADING"? I tried pytorch 1.10和1.13, both didn’t work.  Below is  my environment and test code.</p>
<pre><code class="lang-auto">&gt;&gt;&gt; nsys status -e
Timestamp counter supported: Yes
Sampling Environment Check
Linux Kernel Paranoid Level = 1: OK
Linux Distribution = Ubuntu
Linux Kernel Version = 5.14.0-1056-oem: OK
Linux perf_event_open syscall available: OK
Sampling trigger event available: OK
Intel(c) Last Branch Record support: Available
Sampling Environment: OK
</code></pre>
<pre><code class="lang-auto">import os
import torch
import nvtx
import time
from torch import nn
from torch.optim import SGD
from torch.utils.data import Dataset, DataLoader, DistributedSampler

class TestDataset(Dataset):
    def __init__(self) -&gt; None:
        self.x = torch.randn(num_samples, 10)
        self.y = torch.randn(num_samples, 1)

    # @nvtx.annotate("data loading",color='yellow')
    def __getitem__(self, index):
        torch.cuda.nvtx.range_push("data loading")
        x = self.x[index]
        y = self.y[index]
        time.sleep(0.1)
        torch.cuda.nvtx.range_pop()
        return x,y

    def __len__(self):
        return num_samples

class Model(nn.Module):
    def __init__(self) -&gt; None:
        super(Model, self).__init__()
        self.classifer = nn.Linear(10, 3)
        self.pred = nn.Linear(10, 3)

    # @nvtx.annotate("forward",color='blue')
    def forward(self, x):
        pred = self.pred(x)
        classifier_out = self.classifer(x)
        classifer_score = torch.softmax(classifier_out, dim=1)
        _, index = torch.max(classifer_score, dim=1)
        return pred[torch.arange(index.shape[0]), index]

@nvtx.annotate("train",color='red')
def train():
    model = Model().to(device)
    optimizer = SGD(model.parameters(), 0.1)
    loss_fn = nn.MSELoss()
    dataset = TestDataset()
    # train_sampler = DistributedSampler(dataset)
    dataloader = DataLoader(dataset, batch_size = batch_size, num_workers = 2, sampler=None)#,sampler=train_sampler
    for epoch in range(1):
        for batch, (x, y) in enumerate(dataloader):
            if batch == 1: torch.cuda.cudart().cudaProfilerStart()
            if batch &gt;= 1: torch.cuda.nvtx.range_push("iteration{}".format(batch))

            x = x.to(device)
            y = y.to(device)
            optimizer.zero_grad()

            if batch &gt;= 1: torch.cuda.nvtx.range_push("forward")
            output = model(x)
            if batch &gt;= 1: torch.cuda.nvtx.range_pop()

            loss = loss_fn(output, y)

            if batch &gt;= 1: torch.cuda.nvtx.range_push("backward")
            loss.backward()
            if batch &gt;= 1: torch.cuda.nvtx.range_pop()
            
            if batch &gt;= 1: torch.cuda.nvtx.range_push("opt.step()")
            optimizer.step()
            if batch &gt;= 1: torch.cuda.nvtx.range_pop()

            # print('pred.bias:', model.state_dict()['pred.bias'])
            print('classifer.bias:', model.state_dict()['classifer.bias'])
            if batch &gt;= 1: torch.cuda.nvtx.range_pop()# iteration

    torch.cuda.cudart().cudaProfilerStop()
    

if __name__ == "__main__":
    os.environ['CUDA_LAUNCH_BLOCKING']='1'
    batch_size = 4
    num_samples = 20
    device = torch.device('cuda')
    
    train()
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/2/826f8fd619baa7f13252cbf4e1bc4cff49902d15.png" data-download-href="https://discuss.pytorch.org/uploads/default/826f8fd619baa7f13252cbf4e1bc4cff49902d15" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/826f8fd619baa7f13252cbf4e1bc4cff49902d15_2_690x351.png" alt="image" data-base62-sha1="iBT5F7tugXteEUP9fy5grdnfXJH" width="690" height="351" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/826f8fd619baa7f13252cbf4e1bc4cff49902d15_2_690x351.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/826f8fd619baa7f13252cbf4e1bc4cff49902d15_2_1035x526.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/2/826f8fd619baa7f13252cbf4e1bc4cff49902d15_2_1380x702.png 2x" data-dominant-color="E6E5E6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1498×764 120 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Run code:</p>
<pre><code class="lang-auto">nsys profile -w true -t cuda,nvtx,osrt,cudnn,cublas -s cpu --force-overwrite=true --capture-range=cudaProfilerApi --capture-range-end=stop-shutdown --cudabacktrace=true -x true --output=quickstart python test_nsys.py
</code></pre>

Xinchengzelin: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Hi, I find a way to visu Dataset ‘data loading’ label–add a parameter <code>Dataloader(multiprocessing_context='spawn')</code>,  but this method will improve the memory usage, Is it the only solution to use nvtx in pytorch Dataloader?</p>

5Hyeons: <p>Have you tried setting num_workers to 0?<br>
In my case, default was 2 but when i fixed it works!</p>