PyTorch NCCL Flight Recorder

username: gjit_JNPR
cooked: <p>The recently published LLaMa-3 paper (<a href="https://arxiv.org/pdf/2407.21783" rel="noopener nofollow ugc">https://arxiv.org/pdf/2407.21783</a>) mentions the use of a PyTorch built-in feature called NCCL Flight Recorder as shown below:. On further exploring PyTorch, I did not find any such feature that can record NCCL events.</p>
<p><strong>Question:</strong></p>
<ol>
<li>Is this feature open sourced?</li>
<li>If yes, is there any documentation or summary of it?</li>
</ol>

username: gnadathur
cooked: <p>Thanks for reaching out.<br>
Flight recorder will be released as prototype API in 2.5  release with documentation.<br>
cc: <a class="mention" href="/u/c-p-i-o">@c-p-i-o</a></p>

username: c-p-i-o
cooked: <p>Hey Jit! Yes, flight recorder is open sourced and available in the PyTorch branch. We just haven’t broadly made any announcements as we’ve been testing and changing some of the underlying implementation.</p>
<p>We’re writing up some documentation next few days on how to enable it and make use of the generated data to detect stuck jobs. Please stand by. As <a class="mention" href="/u/gnadathur">@gnadathur</a> mentioned, FR will be marked Prototype in upcoming 2.5 release.</p>

username: gjit_JNPR
cooked: <p>Hi Chiraj and Gokul,</p>
<p>Thanks for your responses! Looking forward to being able to use the NCCL Flight Recorder.</p>

username: Jay_you_Dosen
cooked: <p>Could you please tell me which branch the flight recorder is in? thanks</p>

username: c-p-i-o
cooked: <p>Flight recorder collection portion has been available in 2.4 and in the main branch.<br>
The tool to analyze FR traces is under <code>/tools/flight_recorder/fr_trace.py</code> in the main branch.</p><aside class="onebox githubfolder" data-onebox-src="https://github.com/pytorch/pytorch/tree/main/tools/flight_recorder">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/pytorch/pytorch/tree/main/tools/flight_recorder" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h3><a href="https://github.com/pytorch/pytorch/tree/main/tools/flight_recorder" target="_blank" rel="noopener nofollow ugc">pytorch/tools/flight_recorder at main · pytorch/pytorch</a></h3>


  <p><span class="label1">Tensors and Dynamic neural networks in Python with strong GPU acceleration - pytorch/pytorch</span></p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: c-p-i-o
cooked: <p>Also, I’m finishing a tutorial here. Almost done.</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/tutorials/blob/cpio/flight_recorder/prototype_source/flight_recorder_tutorial.rst">
  <header class="source">

      <a href="https://github.com/pytorch/tutorials/blob/cpio/flight_recorder/prototype_source/flight_recorder_tutorial.rst" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/tutorials/blob/cpio/flight_recorder/prototype_source/flight_recorder_tutorial.rst" target="_blank" rel="noopener nofollow ugc">pytorch/tutorials/blob/cpio/flight_recorder/prototype_source/flight_recorder_tutorial.rst</a></h4>


      <pre><code class="lang-rst">(prototype) Flight Recorder for Debugging
=========================================
**Author**: `Chirag Pandya &lt;https://github.com/c-p-i-o&gt;`, `Junjie Wang &lt;https://github.com/fduwjj&gt;`

This tutorial introduces a new tool for debugging stuck jobs during distributed training.

Background and Motivation
--------------------------
An AI distributed training job refers to the process of training a machine learning model using multiple devices, such
as GPUs or CPUs, connected in a network. This approach allows for faster and more efficient training of large models
that require significant computational resources.
An engineer’s goal is to complete an AI training job as fast as possible and make continuous improvements such that
subsequent training can be done faster. A trained usable model is the final desired outcome.
One of the biggest impediment to completing training is the concept of a "stuck job".

A distributed AI training job is considered "stuck" when it stops making meaningful progress for an extended period of
time.

A job can get stuck for various reasons:
    - Data Starvation: This happens when the training job is not receiving data at the expected rate. This could be due to
</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/tutorials/blob/cpio/flight_recorder/prototype_source/flight_recorder_tutorial.rst" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: Jay_you_Dosen
cooked: <p>Thanks. It’s very helpful. <img src="https://discuss.pytorch.org/images/emoji/apple/ghost.png?v=12" title=":ghost:" class="emoji" alt=":ghost:" loading="lazy" width="20" height="20"></p>

username: jinqinn
cooked: <p>Thanks. its very helpful.</p>

username: c-p-i-o
cooked: <p>Hey <a class="mention" href="/u/jinqinn">@jinqinn</a>, the official tutorial is published <a href="https://pytorch.org/tutorials/prototype/flight_recorder_tutorial.html" rel="noopener nofollow ugc">here</a>. Let us know if you’ve had any success with Flight Recorder.</p>

username: tmm1
cooked: <p>The tutorial says I can run <code>torchfrtrace</code> directly when using nightly, but it didn’t work for me.</p>
<pre><code class="lang-auto">❯ uv run torchfrtrace -h
Traceback (most recent call last):
  File ".venv/bin/torchfrtrace", line 5, in &lt;module&gt;
    from tools.flight_recorder.fr_trace import main
ModuleNotFoundError: No module named 'tools'
</code></pre>

username: danielz
cooked: <p>Yes, I have the same issue. This is probably because we are using a non-nightly build.</p>
<blockquote>
<p>If you install the PyTorch nightly build or build from scratch with USE_DISTRIBUTED=1, you can directly use the following command directly:</p>
</blockquote>
<pre><code class="lang-auto">torchfrtrace &lt;dump dir containing trace files&gt; [-o &lt;output file&gt;]
</code></pre>
<p>EDIT: Actually, I still have the same error even in the nightly build docker.</p>

username: c-p-i-o
cooked: <p>Oops. It seems related to PYTHONPATH not correctly set for the module.</p>
<pre><code class="lang-auto">$pwd
/home/cpio/local/pytorch
$export PYTHONPATH=$(pwd):$PYTHONPATH
$torchfrtrace
Traceback (most recent call last):
  File "/home/cpio/local/b/pytorch-env/bin/torchfrtrace", line 33, in &lt;module&gt;
    sys.exit(load_entry_point('torch', 'console_scripts', 'torchfrtrace')())
  File "/home/cpio/local/pytorch/tools/flight_recorder/fr_trace.py", line 44, in main
    assert args.trace_dir, "Trace directory trace_dir is required"
AssertionError: Trace directory trace_dir is required
</code></pre>
<p>Let me see if I can resolve this.</p>

username: danielz
cooked: <p>Yes, I had to point <code>PYTHONPATH</code> to <code>pytorch/tools</code></p>

username: c-p-i-o
cooked: <p>Thanks for the quick test! I’ll figure out the packaging issue so that you don’t have to set <code>PYTHONPATH</code>.<br>
I quickly tried adding an empty <code>__init__.py</code> file in the tools directory but that didn’t seem to help.</p>