Profiling what happens in thread

Valentin_Thorey: <p>Is there a way to record what happens in a thread triggered inside the profile block ?</p>
<p>For instance</p>
<pre data-code-wrap="python"><code class="lang-python">import threading
import time

import torch
from torch.profiler import record_function


def worker():
    for _ in range(20):
        with record_function("worker_random"):
            torch.rand(2048, 2048)
            time.sleep(0.05)


with torch.profiler.profile(activities=[torch.profiler.ProfilerActivity.CPU]) as p:
    t = threading.Thread(target=worker)
    t.start()
    for _ in range(10):
        with record_function("main_thread_empty"):
            torch.empty(1024, 1024)
            time.sleep(0.1)
    t.join()
p.export_chrome_trace("trace.json")
</code></pre>
<p>In the trace I only see the “main_thread_empty” and the profiler as seen in the following screenshot:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/4/34b45504a210d31bebc9103a23a6383bf4e95724.png" data-download-href="https://discuss.pytorch.org/uploads/default/34b45504a210d31bebc9103a23a6383bf4e95724" title="Screenshot 2025-06-19 at 15.19.13"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/34b45504a210d31bebc9103a23a6383bf4e95724_2_690x229.png" alt="Screenshot 2025-06-19 at 15.19.13" data-base62-sha1="7wfbFtGEYxVgPg3qrYGULvZvtoE" width="690" height="229" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/34b45504a210d31bebc9103a23a6383bf4e95724_2_690x229.png, https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/34b45504a210d31bebc9103a23a6383bf4e95724_2_1035x343.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/4/34b45504a210d31bebc9103a23a6383bf4e95724_2_1380x458.png 2x" data-dominant-color="C8CACB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2025-06-19 at 15.19.13</span><span class="informations">2450×816 50.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Valentin_Thorey: <p>“If multiple profiler ranges are active at the same time (e.g. in parallel PyTorch threads), each profiling context manager tracks only the operators of its corresponding range.”</p>
<p>I found this <a href="https://docs.pytorch.org/tutorials/recipes/recipes/profiler_recipe.html" rel="noopener nofollow ugc">here</a>, which seems to indicate that it may not be possible? I suppose one workaround would be to create a second profiler in the thread and open the trace in two tabs for comparison.</p>

Valentin_Thorey: <p>Actually found the “profile_all_threads” option that can be added, it works well this this, at least for this simple scenario:</p>
<pre><code class="lang-auto">import threading
import time

import torch
from torch.profiler import _ExperimentalConfig, record_function

if __name__ == "__main__":

    def worker():
        for _ in range(20):
            with record_function("worker_random"):
                torch.rand(2048, 2048)
                time.sleep(0.05)

    with torch.profiler.profile(
        activities=[torch.profiler.ProfilerActivity.CPU],
        experimental_config=_ExperimentalConfig(profile_all_threads=True),
    ) as p:
        t = threading.Thread(target=worker)
        t.start()
        for _ in range(10):
            with record_function("main_thread_empty"):
                torch.empty(1024, 1024)

                time.sleep(0.1)
        t.join()
    p.export_chrome_trace("trace.json")
</code></pre>
<p>Will give</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/c/bc8fd3b59eb5e3404a0e92369c9383047684f2d8.png" data-download-href="https://discuss.pytorch.org/uploads/default/bc8fd3b59eb5e3404a0e92369c9383047684f2d8" title="Screenshot 2025-06-23 at 10.26.25"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/c/bc8fd3b59eb5e3404a0e92369c9383047684f2d8_2_690x185.png" alt="Screenshot 2025-06-23 at 10.26.25" data-base62-sha1="qU5Uz4pUHk70ztRmIyrZl99yxlu" width="690" height="185" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/c/bc8fd3b59eb5e3404a0e92369c9383047684f2d8_2_690x185.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/c/bc8fd3b59eb5e3404a0e92369c9383047684f2d8_2_1035x277.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/c/bc8fd3b59eb5e3404a0e92369c9383047684f2d8_2_1380x370.png 2x" data-dominant-color="BEC2C7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2025-06-23 at 10.26.25</span><span class="informations">2514×676 68.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>