Triton kernel launch in TorchInductor

trusira: <p>I’m trying to understand how TorchInductor is scheduling generated Triton kernels for execution. I can see in <a href="https://github.com/pytorch/pytorch/blob/2a3d8e50fb2923990a5549de63ea38a2e0671cc6/torch/_inductor/triton_heuristics.py#L180" rel="noopener nofollow ugc"><code>precompile</code></a> function of <code>CachingAutotuner</code>, kernel binaries and launchers are being populated. But I’m not sure where these launchers are actually launched/called and how corresponding <code>cudaLaunchKernel</code>s are issued.</p>
<p>Could someone please point me in the right direction?<br>
Thanks in advance!</p>

trusira: <p>From what I was able to find, this is handled by Triton during the launcher generation for each kernel in<br>
<a href="https://github.com/openai/triton/blob/0027cf785efa64ff7e6120bf1633b4ce6291c54d/python/triton/compiler/make_launcher.py#L66" rel="noopener nofollow ugc"><code>def generate_launcher(constants, signature, ids)</code></a>.<br>
It compiles the CUDA launcher code into a shared library and commits to the codecache.</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/openai/triton/blob/0027cf785efa64ff7e6120bf1633b4ce6291c54d/python/triton/compiler/make_launcher.py#L39">
  <header class="source">

      <a href="https://github.com/openai/triton/blob/0027cf785efa64ff7e6120bf1633b4ce6291c54d/python/triton/compiler/make_launcher.py#L39" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/openai/triton/blob/0027cf785efa64ff7e6120bf1633b4ce6291c54d/python/triton/compiler/make_launcher.py#L39" target="_blank" rel="noopener nofollow ugc">openai/triton/blob/0027cf785efa64ff7e6120bf1633b4ce6291c54d/python/triton/compiler/make_launcher.py#L39</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="29" style="counter-reset: li-counter 28 ;">
          <li>    # retrieve stub from cache if it exists</li>
          <li>    cache_path = so_cache_manager.get_file(so_name)</li>
          <li>    if cache_path is None:</li>
          <li>        with tempfile.TemporaryDirectory() as tmpdir:</li>
          <li>            src = generate_launcher(constants, signature, ids)</li>
          <li>            src_path = os.path.join(tmpdir, "main.c")</li>
          <li>            with open(src_path, "w") as f:</li>
          <li>                f.write(src)</li>
          <li>            so = _build(name, src_path, tmpdir)</li>
          <li>            with open(so, "rb") as f:</li>
          <li class="selected">                return so_cache_manager.put(f.read(), so_name, binary=True)</li>
          <li>    else:</li>
          <li>        return cache_path</li>
          <li></li>
          <li></li>
          <li># ----- source code generation --------</li>
          <li></li>
          <li></li>
          <li>def ty_to_cpp(ty):</li>
          <li>    if ty[0] == '*':</li>
          <li>        return "hipDeviceptr_t" if is_hip() else "CUdeviceptr"</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>