stacks file from profiler is empty
Labels: oncall: profiler

### 🐛 Describe the bug

When running the following code, the resulting .stacks files are empty.
Also, the tensorboard chart shows 50/50 split between CPU & "other" but no time spent in the GPU.
I suspect this is not a bug, but me mis-using the API. Yet, I'm not sure how.

```python
import torch

class Mlp(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = torch.nn.Linear(100, 100)
        self.act = torch.nn.ReLU()
        self.fc2 = torch.nn.Linear(100, 100)

    def forward(self, x):
        return self.fc2(self.act(self.fc1(x)))

class Net(torch.nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.mlp1 = Mlp()
        self.mlp2 = Mlp()

    def forward(self, x):
        return self.mlp2(self.mlp1(x))

from torch.utils.data import DataLoader
import torchdata.datapipes.iter as dp
from torch.profiler import profile, ProfilerActivity

def make_mock_dataloader():
    pipe = dp.IterableWrapper([torch.rand(100) for _ in range(1000)])
    return DataLoader(pipe, batch_size=32, num_workers=2, drop_last=True)

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
net = Net().to(device=device)

target_traces = 2
traces_saved = 0

def trace_handler(prof: "torch.profiler.profile"):
    global traces_saved
    from os.path import join

    print("SAVING TRACE")

    tb_dir = join("./output2", "traces", str(traces_saved))
    handler = torch.profiler.tensorboard_trace_handler(
        tb_dir, worker_name=f"rank0"
    )
    handler(prof)

    prof.export_stacks(path=join(tb_dir, f"rank0.cuda.stacks"), metric="self_cuda_time_total")
    prof.export_stacks(path=join(tb_dir, f"rank0.cpu.stacks"), metric="self_cpu_time_total")

    # print(prof.events())

    traces_saved += 1
    if traces_saved == target_traces:
        prof.stop()

prof = profile(
    activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
    # profile_memory=True,
    with_stack=True,
    # with_modules=True,
    # record_shapes=True,
    schedule=torch.profiler.schedule(
        skip_first=5, wait=1, warmup=5, active=5, repeat=target_traces
    ),
    on_trace_ready=trace_handler,
)
prof.start()

for idx, batch in enumerate(make_mock_dataloader()):
    print(f"idx: {idx}")
    batch = batch.to(device=device)
    out = net(batch)
    out.sum().backward()
    net.zero_grad()
    prof.step()
```

### Versions

Collecting environment information...
PyTorch version: 1.14.0.dev20221027+cu116
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31

Python version: 3.8.15 (default, Nov  4 2022, 20:59:55)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.15.0-50-generic-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA GeForce GTX 1080 Ti
GPU 1: NVIDIA GeForce GTX 1080 Ti

Nvidia driver version: 515.65.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-boto3-cloudformation==1.25.4
[pip3] mypy-boto3-dynamodb==1.25.0
[pip3] mypy-boto3-ec2==1.25.5
[pip3] mypy-boto3-lambda==1.25.0
[pip3] mypy-boto3-rds==1.25.1
[pip3] mypy-boto3-s3==1.25.0
[pip3] mypy-boto3-sqs==1.25.0
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.4
[pip3] torch==1.14.0.dev20221027+cu116
[pip3] torch-tb-profiler==0.4.0
[pip3] torchdata==0.6.0.dev20221027
[pip3] torchsnapshot-nightly==2022.10.29
[pip3] torchvision==0.15.0a0+edb3a80
[conda] numpy                     1.23.4                   pypi_0    pypi
[conda] torch                     1.14.0.dev20221027+cu116          pypi_0    pypi
[conda] torch-tb-profiler         0.4.0                    pypi_0    pypi
[conda] torchdata                 0.6.0.dev20221027          pypi_0    pypi
[conda] torchsnapshot-nightly     2022.10.29               pypi_0    pypi
[conda] torchvision               0.15.0a0+edb3a80          pypi_0    pypi

cc @robieta @chaekit @aaronenyeshi @ngimel @nbcsm @guotuofeng @guyang3532 @gaoteng-git @tiffzhaofb