Tensorboard and HTA not loading TRACE FILES

Ilex00para: <p>Hello,</p>
<p>I want to trace my model. Started with the Profiling PyTorch Tutorials:</p>
<p>1 Step: trace file is saved in the correct folder.</p>
<pre><code class="lang-auto">with torch.profiler.profile(
        schedule=torch.profiler.schedule(wait=1, warmup=1, active=3, repeat=1), #schedule the profiler to start after 1 step, warmup for 1 step, run for 3 steps and repeat 1 time
        on_trace_ready= torch.profiler.tensorboard_trace_handler('./traces'), #save the trace to tensorboard by using a tracer object
        record_shapes=True,
        profile_memory=True,
        with_stack=True
) as prof:
    for step, batch_data in enumerate(train_loader):
        prof.step()  # Need to call this at each step to notify profiler of steps' boundary.
        if step &gt;= 1 + 1 + 3:
            break
        train(batch_data)
</code></pre>
<p>2 Step (ERROR 1): Displaying the trace file (json) in tensorboard gives the first error. I tried to look in forums which for similar error and found but it did not solve my problem. I tried several paths and can exclude this cause.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/a/fa96c65fdfc86561fefa5c90d92525c075541e8b.png" data-download-href="https://discuss.pytorch.org/uploads/default/fa96c65fdfc86561fefa5c90d92525c075541e8b" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/a/fa96c65fdfc86561fefa5c90d92525c075541e8b.png" alt="image" data-base62-sha1="zKOoSM8hJCXQX7zxzrlvKb66GNR" width="690" height="392" data-dominant-color="3E3F40"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">795×452 40.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>3 Step (ERROR 2): Reading the file (json) with holistic trace analysis.</p>
<pre><code class="lang-auto">2024-08-07 18:05:24,170 - hta - trace.py:L389 - INFO - C:/????/?????/??????/PyTorch/log/resnet18/
2024-08-07 18:05:24,307 - hta - trace_file.py:L61 - ERROR - If the trace file does not have the rank specified in it, then add the following snippet key to the json files to use HTA; "distributedInfo": {"rank": 0}. If there are multiple traces files, then each file should have a unique rank value.
2024-08-07 18:05:24,447 - hta - trace_file.py:L61 - ERROR - If the trace file does not have the rank specified in it, then add the following snippet key to the json files to use HTA; "distributedInfo": {"rank": 0}. If there are multiple traces files, then each file should have a unique rank value.
2024-08-07 18:05:24,448 - hta - trace_file.py:L92 - WARNING - There is no item in the rank to trace file map.
2024-08-07 18:05:24,448 - hta - trace.py:L535 - INFO - ranks=[]
2024-08-07 18:05:24,449 - hta - trace.py:L541 - ERROR - The list of ranks to be parsed is empty.
</code></pre>
<p>Questions:</p>
<ol>
<li>Why is the rank missing? Is this the right file?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/4/44c5adff372ae39ef104ebb4e87fd7f627d4848b.png" data-download-href="https://discuss.pytorch.org/uploads/default/44c5adff372ae39ef104ebb4e87fd7f627d4848b" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/4/44c5adff372ae39ef104ebb4e87fd7f627d4848b.png" alt="image" data-base62-sha1="9OnZFMT6tqFNLB75JxiJkAe1gH9" width="690" height="182" data-dominant-color="212222"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1287×340 8.77 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></li>
<li>I saw multiple notation that tensorboard is deprecated and hta is now prefered? Is there a difference between their trace files? Do I understand something wrong? (In the hta documentation it refers to the same code I have used.</li>
</ol>
<p>Thank you in advance.</p>

Ilex00para: <p>I solved the problem with HTA.</p>
<p>The case is that HTA is more for distributed jobs starting with a simple example is not covered. The rank needs to be specified manually if it is only run on one GPU:  <code>ERROR - If the trace file does not have the rank specified in it, then add the following snippet key to the json files to use HTA; "distributedInfo": {"rank": 0}. If there are multiple traces files, then each file should have a unique rank value.</code>. Which makes it possible to read the file.</p>
<aside class="onebox githubissue" data-onebox-src="https://github.com/facebookresearch/HolisticTraceAnalysis/issues/104">
  <header class="source">

      <a href="https://github.com/facebookresearch/HolisticTraceAnalysis/issues/104" target="_blank" rel="noopener nofollow ugc">github.com/facebookresearch/HolisticTraceAnalysis</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/facebookresearch/HolisticTraceAnalysis/issues/104" target="_blank" rel="noopener nofollow ugc">HTA expects rank to be specified in a trace file</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2024-02-21" data-time="18:03:53" data-timezone="UTC">06:03PM - 21 Feb 24 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2024-02-22" data-time="03:07:09" data-timezone="UTC">03:07AM - 22 Feb 24 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/lishen" target="_blank" rel="noopener nofollow ugc">
          <img alt="lishen" src="https://avatars.githubusercontent.com/u/874315?v=4" class="onebox-avatar-inline" width="20" height="20">
          lishen
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          question
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">### 🐛 Describe the bug

tried to run a simple example to use HTA for the first t<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">ime but ran into error when reading the trace analysis. here's the error message:

&gt; 2024-02-21 12:43:20,242 - hta - trace.py:L389 - INFO - path_to/traces-1
2024-02-21 12:43:20,283 - hta - trace_file.py:L61 - ERROR - If the trace file does not have the rank specified in it, then add the following snippet key to the json files to use HTA; "distributedInfo": {"rank": 0}. If there are multiple traces files, then each file should have a unique rank value.
2024-02-21 12:43:20,283 - hta - trace_file.py:L92 - WARNING - There is no item in the rank to trace file map.
2024-02-21 12:43:20,284 - hta - trace.py:L535 - INFO - ranks=[]
2024-02-21 12:43:20,285 - hta - trace.py:L541 - ERROR - The list of ranks to be parsed is empty.

### Steps to reproduce

first, i created a trace file:
```python
import torch
from torch.profiler import (
    profile, schedule, 
    tensorboard_trace_handler, 
    ProfilerActivity)
import torchvision.models as models

model = models.resnet18().cuda()
inputs = torch.randn(5, 3, 224, 224).cuda()

tracing_schedule = schedule(skip_first=5, wait=5, warmup=2, active=2, repeat=1)
trace_handler = tensorboard_trace_handler(dir_name="./traces-1/", use_gzip=True)

with profile(
    activities = [ProfilerActivity.CPU, ProfilerActivity.CUDA],
    schedule = tracing_schedule,
    on_trace_ready = trace_handler,
    profile_memory = True,
    record_shapes = True,
    with_stack = True
) as prof:

    for idx in range(25):
        model(inputs)
        prof.step()
```
then i tried to read the trace:
```python
from hta.trace_analysis import TraceAnalysis
trace_dir = "./traces-1/"
analyzer = TraceAnalysis(trace_dir=trace_dir)
```

### Expected behavior

`TraceAnalysis` reads the trace file without error so that i can do something cool.

### Environment

OS version: CentOS Linux release 7.9.2009
Python version: 3.9.1
PyTorch version: 2.2.0
torch-tb-profiler: 0.4.3
HTA version: 0.2.0
How did you installed HTA (pip, source): pip

### Additional Info

_No response_</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<aside class="onebox githubissue" data-onebox-src="https://github.com/facebookresearch/HolisticTraceAnalysis/issues/102">
  <header class="source">

      <a href="https://github.com/facebookresearch/HolisticTraceAnalysis/issues/102" target="_blank" rel="noopener nofollow ugc">github.com/facebookresearch/HolisticTraceAnalysis</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/facebookresearch/HolisticTraceAnalysis/issues/102" target="_blank" rel="noopener nofollow ugc">single GPU support</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2024-02-01" data-time="11:17:31" data-timezone="UTC">11:17AM - 01 Feb 24 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2024-02-01" data-time="16:24:41" data-timezone="UTC">04:24PM - 01 Feb 24 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/AwePhD" target="_blank" rel="noopener nofollow ugc">
          <img alt="AwePhD" src="https://avatars.githubusercontent.com/u/93187038?v=4" class="onebox-avatar-inline" width="20" height="20">
          AwePhD
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          help wanted
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          question
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          needs triage
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">### What is your question?

My question is about the direction of the pytorch pr<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">ofiling. To summarize, profiler trace (from meta's kineto) was (and still is) collected by pytorch profiler. Then these traces were input to tensorboard. tensorboard had some flaws - its constrained usages, cannot be scripted to process traces manually - and got deprecated. Then pytorch made an announcement ([1 year ago](https://pytorch.org/blog/trace-analysis-for-masses/)) of opening their traces analysis tool (HTA) source. Originally, it was used for LLM models and it seems to be heavily oriented to distributed profiling - all the performance problem generated by the introduction of distributed computation. That's great and promising, I mean it.

Although, a lot of researchers and deep learning enthusiasts still use a single GPU during their experiments and activities. Since HTA becomes the canonical way to work with the Kineto traces, do you plan to support traces from "not distributed" computation? Namely, without a workaround that implies to add the rank info in the `json` trace file.

Thanks for your work and your time reading this (lenghty) question.

### Code

I am fully aware of the [`hta.common.trace_file:create_rank_to_trace_dict`](https://github.com/facebookresearch/HolisticTraceAnalysis/blob/007e331de8b211fbc5f1c8575486d56d5cfbbb9e/hta/common/trace_file.py#L62C20-L65C18).

### What have you tried?

_No response_

### Environment

_No response_</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Still I am always a reluctant to change something manually in a computer created file and follow up code is not working if the job is not distributed.</p>
<p>PS: This note in the " PyTorch Profiler With TensorBoard" is super confusing</p>
<p>Note<br>
TensorBoard Plugin support has been deprecated, so some of these functions may not work as previously. Please take a look at the replacement, <a href="https://github.com/pytorch/kineto/tree/main#holistic-trace-analysis" rel="noopener nofollow ugc">HTA</a>.</p>