Printing C++ stack traces

tushar00jain: <p>How do we get C++ to print stack traces? I’m getting the following error from <code>ProcessGroupNCCL.cpp</code></p>
<p><code>vector::_M_range_check: __n (which is 1044) &gt;= this-&gt;size() (which is 0)</code></p>
<p>Doing <code>TORCH_SHOW_CPP_STACKTRACES=1</code> doesn’t help print the stack trace.</p>

Ahsen_Zuhal: <p>You can use the built-in GNU backtrace mechanism:</p>
<pre><code class="lang-auto">export TORCH_SHOW_CPP_STACKTRACES=1
export TORCH_CPP_DEBUG=1
</code></pre>

tushar00jain: <p>I still don’t get anything. Particularly, this is the error.</p>
<p>[ProcessGroupNCCL.cpp:2092] Process group watchdog thread terminated with exception: vector::_M_range_check: __n (which is 1049) &gt;= this-&gt;size() (which is 0)</p>
<p>The code only prints <code>what()</code> on the exception. And no stack trace gets printed, just the error message.</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/70ec464c1608116df6d379e097f9149b22407456/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp#L2091">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/70ec464c1608116df6d379e097f9149b22407456/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp#L2091" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/70ec464c1608116df6d379e097f9149b22407456/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp#L2091" target="_blank" rel="noopener nofollow ugc">torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp</a></h4>

<div class="git-blob-info">
  <a href="https://github.com/pytorch/pytorch/blob/70ec464c1608116df6d379e097f9149b22407456/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp#L2091" rel="noopener nofollow ugc"><code>70ec464c1</code></a>
</div>



    <pre class="onebox"><code class="lang-cpp">
      <ol class="start lines" start="2081" style="counter-reset: li-counter 2080 ;">
          <li>  VLOG(2)</li>
          <li>      &lt;&lt; pg_-&gt;logPrefix()</li>
          <li>      &lt;&lt; "main process destroyed cuda before watchdog loop exited, terminating watchdog."</li>
          <li>      &lt;&lt; " (Watchdog caught exception: " &lt;&lt; e.what();</li>
          <li></li>
          <li>} else {</li>
          <li>  // Append error message reported from runLoop</li>
          <li>  const auto exitMsg = c10::str(</li>
          <li>      pg_-&gt;logPrefix(),</li>
          <li>      "Process group watchdog thread terminated with exception: ",</li>
          <li class="selected">      e.what());</li>
          <li>  LOG(ERROR) &lt;&lt; exitMsg;</li>
          <li>  if (C10_LIKELY(rethrowCUDAErrors_) ||</li>
          <li>      !(std::string(e.what()).find("CUDA Error"))) {</li>
          <li>    // TODO(whc) clean up the rethrow - why is it stored in a class var and</li>
          <li>    // rethrown?</li>
          <li>    watchDogException_ =</li>
          <li>        std::make_exception_ptr(C10_BUILD_ERROR(DistBackendError, exitMsg));</li>
          <li>    std::rethrow_exception(watchDogException_);</li>
          <li>  }</li>
          <li>}</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>