Repository: triton-lang/triton
Pull request #6394: [NVIDIA] Enable Programmatic Dependent Launch in Triton
Author: login: depaulmillz
Created: 2025-04-03T22:38:51Z | Merged: 2025-04-29T16:36:22Z

Programmatic Dependent Launch (PDL) enables kernels within the same CUDA stream to overlap while programmatically resolving inter-kernel dependencies. This allows consecutive kernels to overlap their ramp-down and ramp-up periods, efficiently hiding prologue latencies. Inter-kernel dependencies are resolved using Grid Dependency Control (GDC), which ensures that a kernel waits before reading memory written by the preceding kernel. This feature is utilized in libraries including [CUTLASS](https://github.com/NVIDIA/cutlass/blob/main/media/docs/cpp/dependent_kernel_launch.md).

Effectively utilizing PDL in Triton requires using `tl.extra.cuda.gdc_wait()` to wait for the prior kernel to finish writing its results. The most straightforward approach is to execute `tl.extra.cuda.gdc_wait()` before any `tl.load`, based on the conservative assumption that the prior kernel may be launched with PDL and can write to any memory location.

When using PDL, `tl.extra.cuda.gdc_launch_dependents()` allows for the current kernel to trigger the next kernel to start. See the [CUDA documentation](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#programmatic-dependent-launch-and-synchronization) for more information.

We utilize this feature in a simple non-persistent kernel with a conservative approach to inter-kernel dependencies on Blackwell in tutorial 11. This kernel achieves up to a 15% speedup:
![pdl_performance](https://github.com/user-attachments/assets/5d0a9a3b-38f6-4ae1-9a94-7ade22099a4f)

More advanced patterns with PDL we can achieve up to 33% performance benefits on back-to-back layers in LLMs (see [_LLM Inference Performance and Optimization on NVIDIA GB200 NVL72_](https://www.nvidia.com/en-us/on-demand/session/gtc25-s72503/) at GTC 2025 for more details). 



Diff:
diff --git a/docs/index.rst b/docs/index.rst
index e9cf1e79f1cf..7530931695db 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -26,6 +26,7 @@ Python API
 - :doc:`triton.language <python-api/triton.language>`
 - :doc:`triton.testing <python-api/triton.testing>`
 - :doc:`Triton semantics <python-api/triton-semantics>`
+- :doc:`triton.language.extra.cuda <python-api/triton.language.extra.cuda>`
 
 
 .. toctree::
diff --git a/docs/python-api/triton.language.extra.cuda.rst b/docs/python-api/triton.language.extra.cuda.rst
new file mode 100644
index 000000000000..07b930519b57
--- /dev/null
+++ b/docs/python-api/triton.language.extra.cuda.rst
@@ -0,0 +1,14 @@
+triton.language.extra.cuda
+==========================
+
+.. currentmodule:: triton.language.extra.cuda
+
+Programmatic Dependent Launch
+-----------------------------
+
+.. autosummary::
+    :toctree: generated
+    :nosignatures:
+
+    gdc_wait
+    gdc_launch_dependents
diff --git a/python/tutorials/11-programmatic-dependent-launch.py b/python/tutorials/11-programmatic-dependent-launch.py
new file mode 100644
index 000000000000..5d7eeb02647e
--- /dev/null
+++ b/python/tutorials/11-programmatic-dependent-launch.py
@@ -0,0 +1,116 @@
+"""
+Programmatic Dependent Launch
+=====================
+This script demonstrates the use of programmatic dependent launch (PDL) ontop of the vector-add example using Triton.
+
+For CUDA reference on programmatic dependent launch see https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#programmatic-dependent-launch-and-synchronization.
+For PTX reference on programmatic dependent launch see https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-griddepcontrol.
+
+.. code-block:: bash
+    python 11-programmatic-dependent-launch.py
+"""
+
+import torch
+import triton
+import triton.language as tl
+
+
+def is_cuda():
+    return triton.runtime.driver.active.get_current_target().backend == "cuda"
+
+
+def supports_pdl():
+    return is_cuda() and torch.cuda.get_device_capability()[0] >= 9
+
+
+# In this example
+@triton.jit
+def add_kernel(x_ptr,  #
+               y_ptr,  #
+               output_ptr,  #
+               n_elements,  #
+               BLOCK_SIZE: tl.constexpr,  #
+               USE_GDC: tl.constexpr,  #
+               ):
+    pid = tl.program_id(axis=0)
+    block_start = pid * BLOCK_SIZE
+    offsets = block_start + tl.arange(0, BLOCK_SIZE)
+    mask = offsets < n_elements
+    if USE_GDC:
+        # GDC wait waits for ALL programs in the the prior kernel to complete before continuing.
+        # This ensures any memory operations happen before the wait in program order,
+        # e.g. if the prior kernel writes to x or y the new values will be visible.
+        tl.extra.cuda.gdc_wait()
+
+    x = tl.load(x_ptr + offsets, mask=mask)
+    y = tl.load(y_ptr + offsets, mask=mask)
+    if USE_GDC:
+        # GDC launch dependents hints the runtime system to launch dependent kernels.
+        # These dependent kernels must also be launched with PDL enabled.
+        # Once GDC launch has been issued by ALL programs or
+        # programs have finished, the dependent grid can begin if there are enough resources.
+        # Note: this by itself provides no additional memory-ordering guarentees, unlike `gdc_wait`
+        tl.extra.cuda.gdc_launch_dependents()
+    output = x + y
+    tl.store(output_ptr + offsets, output, mask=mask)
+
+
+def add(x: torch.Tensor, y: torch.Tensor, launch_pdl: bool = True):
+    output = torch.empty_like(x)
+    assert x.device == y.device and output.device == x.device
+    n_elements = output.numel()
+    grid = lambda meta: (triton.cdiv(n_elements, meta['BLOCK_SIZE']), )
+    add_kernel[grid](
+        x, y, output, n_elements, BLOCK_SIZE=1024,
+        USE_GDC=launch_pdl,  # set constexpr in kernel to use grid dependence control
+        launch_pdl=launch_pdl,  # launch kernel with PDL flag set enabled
+    )
+    return output
+
+
+def validate(n_elements):
+    x = torch.rand(n_elements, device="cuda", dtype=torch.float32)
+    y = torch.rand(n_elements, device="cuda", dtype=torch.float32)
+
+    torch_result = x + y
+    add_result = add(x, y)
+
+    torch_vs_add = "✅" if torch.allclose(torch_result, add_result, atol=1.0) else "❌"
+    print(f"Number of Elements={n_elements} verification naive vs: ", end="")
+    print(f"add: {torch_vs_add}")
+
+
+@triton.testing.perf_report(
+    triton.testing.Benchmark(
+        x_names=["size"],
+        x_vals=[2**i for i in range(23, 28, 1)],
+        x_log=False,
+        line_arg="provider",
+        line_vals=["pdl-fp32", "fp32"],
+        line_names=["PDL", "No PDL"],
+        styles=[("red", "-"), ("blue", "-")],
+        ylabel='GB/s',
+        plot_name="pdl-performance",
+        args={},
+    ))
+def benchmark(size, provider):
+    x = torch.rand(size, device="cuda", dtype=torch.float32)
+    y = torch.rand(size, device="cuda", dtype=torch.float32)
+
+    quantiles = [0.5, 0.2, 0.8]
+
+    fn = lambda: add(x, y, "pdl" in provider)
+
+    ms, min_ms, max_ms = triton.testing.do_bench_cudagraph(fn, quantiles=quantiles, rep=100)
+
+    gbps = lambda ms: 3 * x.numel() * x.element_size() * 1e-9 / (ms * 1e-3)
+    return gbps(ms), gbps(max_ms), gbps(min_ms)
+
+
+if __name__ == "__main__":
+
+    if supports_pdl():
+        validate(1024)
+        benchmark.run(print_data=True, show_plots=True, save_path=".")
+    else:
+        print("PDL is not supported on this device")
diff --git a/third_party/nvidia/backend/compiler.py b/third_party/nvidia/backend/compiler.py
index 073890a0d248..9487f261d61e 100644
--- a/third_party/nvidia/backend/compiler.py
+++ b/third_party/nvidia/backend/compiler.py
@@ -108,6 +108,7 @@ class CUDAOptions:
     ptx_version: int = None
     enable_fp_fusion: bool = True
     launch_cooperative_grid: bool = False
+    launch_pdl: bool = False
     supported_fp8_dtypes: Tuple[str] = ("fp8e5", "fp8e4b15")
     deprecated_fp8_dtypes: Tuple[str] = ()
     default_dot_input_precision: str = "tf32"
diff --git a/third_party/nvidia/backend/driver.py b/third_party/nvidia/backend/driver.py
index 80b733d63c38..bec45febfa13 100644
--- a/third_party/nvidia/backend/driver.py
+++ b/third_party/nvidia/backend/driver.py
@@ -122,6 +122,9 @@ def ty_to_cpp(ty):
     }[ty]
 
 
+_BASE_ARGS_FORMAT = "iiiKKppOOOOO"
+
+
 def make_launcher(constants, signature):
 
     def _expand_signature(sig, output):
@@ -184,7 +187,7 @@ def format_of(ty):
     signature = {i: s for i, s in enumerate(expand_signature)}
 
     args_format = ''.join([format_of(ty) for ty in signature.values()])
-    format = "iiiKKpOOOOO" + args_format
+    format = _BASE_ARGS_FORMAT + args_format
 
     flat_signature = []
     for sig in signature.values():
@@ -264,67 +267,65 @@ def format_of(ty):
   return cuLaunchKernelExHandle;
 }}
 
-static void _launch(int gridX, int gridY, int gridZ, int num_warps, int num_ctas, int launch_cooperative_grid, int clusterDimX, int clusterDimY, int clusterDimZ, int shared_memory, CUstream stream, CUfunction function, CUdeviceptr global_scratch{', ' + arg_decls if len(arg_decls) > 0 else ''}) {{
+static void _launch(int gridX, int gridY, int gridZ, int num_warps, int num_ctas, int launch_cooperative_grid, int launch_pdl, int clusterDimX, int clusterDimY, int clusterDimZ, int shared_memory, CUstream stream, CUfunction function, CUdeviceptr global_scratch{', ' + arg_decls if len(arg_decls) > 0 else ''}) {{
   void *params[] = {{ {', '.join(params)} }};
   if (gridX*gridY*gridZ > 0) {{
-    if ((num_ctas == 1) && (0 == launch_cooperative_grid)) {{
-      CUDA_CHECK(cuLaunchKernel(function, gridX, gridY, gridZ, 32*num_warps, 1, 1, shared_memory, stream, params, 0));
-    }} else if ((num_ctas == 1) && (0 != launch_cooperative_grid)) {{
-      CUlaunchAttribute launchAttr[1];
+    // 4 attributes that we can currently pass maxmimum
+    CUlaunchAttribute launchAttr[4];
+    static cuLaunchKernelEx_t cuLaunchKernelExHandle = NULL;
+    if (cuLaunchKernelExHandle == NULL) {{
+      cuLaunchKernelExHandle = getLaunchKernelExHandle();
+    }}
+    CUlaunchConfig config;
+    config.gridDimX = gridX;
+    config.gridDimY = gridY;
+    config.gridDimZ = gridZ;
+
+    if (num_ctas != 1) {{
+      config.gridDimX *= clusterDimX;
+      config.gridDimY *= clusterDimY;
+      config.gridDimZ *= clusterDimZ;
+    }}
+
+    config.blockDimX = 32 * num_warps;
+    config.blockDimY = 1;
+    config.blockDimZ = 1;
+    config.sharedMemBytes = shared_memory;
+    config.hStream = stream;
+    config.attrs = launchAttr;
+    int num_attrs = 0;
+
+    if (launch_pdl != 0) {{
+      CUlaunchAttribute pdlAttr = {{ .id = CU_LAUNCH_ATTRIBUTE_PROGRAMMATIC_STREAM_SERIALIZATION, .value = 1}};
+      launchAttr[num_attrs] = pdlAttr;
+      ++num_attrs;
+    }}
+
+    if (launch_cooperative_grid != 0) {{
       CUlaunchAttribute coopAttr = {{ .id = CU_LAUNCH_ATTRIBUTE_COOPERATIVE, .value = 1}};
-      launchAttr[0] = coopAttr;
-
-      CUlaunchConfig config;
-      config.gridDimX = gridX;
-      config.gridDimY = gridY;
-      config.gridDimZ = gridZ;
-      config.blockDimX = 32 * num_warps;
-      config.blockDimY = 1;
-      config.blockDimZ = 1;
-      config.sharedMemBytes = shared_memory;
-      config.hStream = stream;
-      config.attrs = launchAttr;
-      config.numAttrs = 1;
-
-      static cuLaunchKernelEx_t cuLaunchKernelExHandle = NULL;
-      if (cuLaunchKernelExHandle == NULL) {{
-        cuLaunchKernelExHandle = getLaunchKernelExHandle();
-      }}
-      CUDA_CHECK(cuLaunchKernelExHandle(&config, function, params, 0));
-
-    }} else {{
-      CUlaunchAttribute launchAttr[3];
-      launchAttr[0].id = CU_LAUNCH_ATTRIBUTE_CLUSTER_DIMENSION;
-      launchAttr[0].value.clusterDim.x = clusterDimX;
-      launchAttr[0].value.clusterDim.y = clusterDimY;
-      launchAttr[0].value.clusterDim.z = clusterDimZ;
-      launchAttr[1].id = CU_LAUNCH_ATTRIBUTE_CLUSTER_SCHEDULING_POLICY_PREFERENCE;
-      launchAttr[1].value.clusterSchedulingPolicyPreference = CU_CLUSTER_SCHEDULING_POLICY_SPREAD;
-
-      unsigned numAttrs = 2;
-      if (0 != launch_cooperative_grid) {{
-        CUlaunchAttribute coopAttr = {{ .id = CU_LAUNCH_ATTRIBUTE_COOPERATIVE, .value = 1}};
-        launchAttr[2] = coopAttr;
-        numAttrs = 3;
-      }}
-
-      CUlaunchConfig config;
-      config.gridDimX = gridX * clusterDimX;
-      config.gridDimY = gridY * clusterDimY;
-      config.gridDimZ = gridZ * clusterDimZ;
-      config.blockDimX = 32 * num_warps;
-      config.blockDimY = 1;
-      config.blockDimZ = 1;
-      config.sharedMemBytes = shared_memory;
-      config.hStream = stream;
-      config.attrs = launchAttr;
-      config.numAttrs = numAttrs;
-      static cuLaunchKernelEx_t cuLaunchKernelExHandle = NULL;
-      if (cuLaunchKernelExHandle == NULL) {{
-        cuLaunchKernelExHandle = getLaunchKernelExHandle();
-      }}
-      CUDA_CHECK(cuLaunchKernelExHandle(&config, function, params, 0));
+      launchAttr[num_attrs] = coopAttr;
+      ++num_attrs;
+    }}
+
+    if (num_ctas != 1) {{
+      CUlaunchAttribute clusterAttr = {{}};
+      clusterAttr.id = CU_LAUNCH_ATTRIBUTE_CLUSTER_DIMENSION;
+      clusterAttr.value.clusterDim.x = clusterDimX;
+      clusterAttr.value.clusterDim.y = clusterDimY;
+      clusterAttr.value.clusterDim.z = clusterDimZ;
+      launchAttr[num_attrs] = clusterAttr;
+      ++num_attrs;
+
+      CUlaunchAttribute clusterSchedulingAttr = {{}};
+      clusterSchedulingAttr.id = CU_LAUNCH_ATTRIBUTE_CLUSTER_SCHEDULING_POLICY_PREFERENCE;
+      clusterSchedulingAttr.value.clusterSchedulingPolicyPreference = CU_CLUSTER_SCHEDULING_POLICY_SPREAD;
+      launchAttr[num_attrs] = clusterSchedulingAttr;
+      ++num_attrs;
     }}
+
+    config.numAttrs = num_attrs;
+
+    CUDA_CHECK(cuLaunchKernelExHandle(&config, function, params, 0));
   }}
 }}
 
@@ -444,6 +445,7 @@ def format_of(ty):
   uint64_t _stream;
   uint64_t _function;
   int launch_cooperative_grid;
+  int launch_pdl;
   PyObject *launch_enter_hook = NULL;
   PyObject *launch_exit_hook = NULL;
   PyObject *kernel_metadata = NULL;
@@ -451,7 +453,7 @@ def format_of(ty):
   PyObject *global_scratch_obj = NULL;
   {newline.join([f"{_extracted_type(ty)} _arg{i};" for i, ty in signature.items()])}
   if(!PyArg_ParseTuple(args, \"{format}\", &gridX, &gridY, &gridZ,
-                                           &_stream, &_function, &launch_cooperative_grid, &global_scratch_obj,
+                                           &_stream, &_function, &launch_cooperative_grid, &launch_pdl, &global_scratch_obj,
                                            &kernel_metadata, &launch_metadata,
                                            &launch_enter_hook, &launch_exit_hook{args_list})) {{
     return NULL;
@@ -485,7 +487,7 @@ def format_of(ty):
   {newline.join(ptr_decls)}
   {newline.join(tma_decls)}
   Py_BEGIN_ALLOW_THREADS;
-  _launch(gridX, gridY, gridZ, num_warps, num_ctas, launch_cooperative_grid, clusterDimX, clusterDimY, clusterDimZ, shared_memory, (CUstream)_stream, (CUfunction)_function, global_scratch{', ' + ', '.join(internal_args_list) if len(internal_args_list) > 0 else ''});
+  _launch(gridX, gridY, gridZ, num_warps, num_ctas, launch_cooperative_grid, launch_pdl, clusterDimX, clusterDimY, clusterDimZ, shared_memory, (CUstream)_stream, (CUfunction)_function, global_scratch{', ' + ', '.join(internal_args_list) if len(internal_args_list) > 0 else ''});
   Py_END_ALLOW_THREADS;
   if (PyErr_Occurred()) {{
     return NULL;
@@ -584,8 +586,8 @@ def wrap_handle_tensordesc(launcher, tensordesc_meta):
         return launcher
 
     def inner(*args):
-        meta_args = args[:11]
-        raw_kernel_args = args[11:]
+        meta_args = args[:len(_BASE_ARGS_FORMAT)]
+        raw_kernel_args = args[len(_BASE_ARGS_FORMAT):]
         tensordesc_idx = 0
         final_args = []
         for i, arg in enumerate(raw_kernel_args):
@@ -619,6 +621,7 @@ def __init__(self, src, metadata):
         self.global_scratch_size = metadata.global_scratch_size
         self.global_scratch_align = metadata.global_scratch_align
         self.launch_cooperative_grid = metadata.launch_cooperative_grid
+        self.launch_pdl = metadata.launch_pdl
 
     def __call__(self, gridX, gridY, gridZ, stream, function, *args):
         if self.global_scratch_size > 0:
@@ -627,7 +630,8 @@ def __call__(self, gridX, gridY, gridZ, stream, function, *args):
             global_scratch = _allocation._allocator(alloc_size, self.global_scratch_align, stream)
         else:
             global_scratch = None
-        self.launch(gridX, gridY, gridZ, stream, function, self.launch_cooperative_grid, global_scratch, *args)
+        self.launch(gridX, gridY, gridZ, stream, function, self.launch_cooperative_grid, self.launch_pdl,
+                    global_scratch, *args)
 
 
 class CudaDriver(GPUDriver):
diff --git a/third_party/nvidia/language/cuda/__init__.py b/third_party/nvidia/language/cuda/__init__.py
index 62edc5d0de47..fbececf1defc 100644
--- a/third_party/nvidia/language/cuda/__init__.py
+++ b/third_party/nvidia/language/cuda/__init__.py
@@ -1,6 +1,7 @@
 from . import libdevice
 
 from .utils import (globaltimer, num_threads, num_warps, smid, convert_custom_float8_sm70, convert_custom_float8_sm80)
+from .gdc import (gdc_launch_dependents, gdc_wait)
 
 __all__ = [
     "libdevice",
@@ -10,4 +11,6 @@
     "smid",
     "convert_custom_float8_sm70",
     "convert_custom_float8_sm80",
+    "gdc_launch_dependents",
+    "gdc_wait",
 ]
diff --git a/third_party/nvidia/language/cuda/gdc.py b/third_party/nvidia/language/cuda/gdc.py
new file mode 100644
index 000000000000..41b9e3743303
--- /dev/null
+++ b/third_party/nvidia/language/cuda/gdc.py
@@ -0,0 +1,42 @@
+"""
+Grid Dependency Control (GDC) is a mechanism used when enabling programmatic dependent launch to launch and
+synchronize grids. These APIs expose GDC to the programmer.
+
+Programmatic dependent launch is supported on SM90 (Hopper) and beyond.
+For PTX reference on grid dependency control see https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-griddepcontrol.
+"""
+
+from triton.language import core
+
+
+@core.extern
+def gdc_wait(_builder=None):
+    """
+    GDC wait is a blocking instruction that waits for all instructions in a prior kernel to complete before continuing.
+    This ensures all memory operations happening before the wait is visible to instructions after it,
+    e.g. if the prior kernel writes to address "x" the new values will be visible in this kernel after the wait.
+
+    This instruction is also safe to execute when programatic dependent launch is disabled.
+
+    See https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-griddepcontrol for more details.
+    """
+    core.inline_asm_elementwise("griddepcontrol.wait; // dummy $0", "=r", [], dtype=core.int32, is_pure=False, pack=1,
+                                _builder=_builder)
+
+
+@core.extern
+def gdc_launch_dependents(_builder=None):
+    """
+    This operation when launched with programmatic dependent launch signals that
+    the next program may launch once all programs in the current kernel
+    call this function or complete.
+
+    Repeated calls to this function have no effect past the first call, and the first call should be
+    treated by the programmer as a hint to the runtime system to launch the next kernel.
+
+    This instruction is also safe to execute when programatic dependent launch is disabled.
+
+    See https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-griddepcontrol for more details.
+    """
+    core.inline_asm_elementwise("griddepcontrol.launch_dependents; // dummy $0", "=r", [], dtype=core.int32,
+                                is_pure=False, pack=1, _builder=_builder)


Reviews:
user: login: Mogball
body: Unrelated to this PR, but why can't we also have an nvgpu pass pessimistically insert gdc_wait() and gdc_launcu_dependents() at the start and end of each kernel if they were not added by a user? Most kernels have some amount of code before the first access to global memory. 
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-03T22:43:18Z
user: login: peterbell10
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-03T22:53:35Z
user: login: depaulmillz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-07T21:48:53Z
user: login: peterbell10
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T20:17:06Z
user: login: peterbell10
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T20:26:31Z
user: login: depaulmillz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T20:37:46Z
user: login: peterbell10
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-09T21:23:06Z
user: login: ThomasRaoux
body: Couple minor comments.

Is there a way to add a unit test for this feature? I guess it is hard to verify that it does something besides performance.
I don't know if tutorial is the right thing do, we don't have microbenchmark so that's probably fine but it would be nice to minimize the tutorial to make it simpler to use. Does this even need to be a matmul?
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-10T01:36:07Z
user: login: depaulmillz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-11T00:40:01Z
user: login: depaulmillz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-11T00:41:42Z
user: login: depaulmillz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-11T17:01:45Z
user: login: depaulmillz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-11T17:03:16Z
user: login: ThomasRaoux
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-11T17:15:11Z
user: login: depaulmillz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T16:38:14Z
user: login: depaulmillz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T16:39:03Z
user: login: peterbell10
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T17:23:33Z
user: login: depaulmillz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T21:09:21Z
user: login: depaulmillz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T21:10:58Z
user: login: depaulmillz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-16T21:11:31Z
user: login: ThomasRaoux
body: LGTM but please wait for @peterbell10 to confirm that all his comments are addressed before landing.
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-17T09:44:19Z
user: login: peterbell10
body: Code looks great, just some concerns about documentation.
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-19T13:09:53Z
user: login: peterbell10
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-19T13:12:26Z
user: login: peterbell10
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-19T17:23:05Z
user: login: peterbell10
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-23T21:54:59Z
user: login: depaulmillz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-29T02:51:57Z
user: login: depaulmillz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-29T02:52:09Z
user: login: peterbell10
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-29T11:45:16Z
user: login: depaulmillz
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-29T15:43:31Z
user: login: peterbell10
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-29T16:06:20Z

Review comments:
diff_hunk: @@ -0,0 +1,262 @@
+"""
+Programmatic Dependent Launch
+=====================
+This script demonstrates the use of programmatic dependent launch ontop of a non-persistent implementation of matrix multiplication using Triton.
+The kernels support both FP16 and FP8 data types but the FP8 implementation is only available on CUDA devices with compute capability >= 9.0.
+
+.. code-block:: bash
+    python 11-programmatic-dependent-launch.py
+"""
+
+import torch
+import triton
+import triton.language as tl
+
+from typing import Optional
+
+
+def is_cuda():
+    return triton.runtime.driver.active.get_current_target().backend == "cuda"
+
+
+def supports_tma():
+    return is_cuda() and torch.cuda.get_device_capability()[0] >= 9
+
+
+def _matmul_launch_metadata(grid, kernel, args):
+    ret = {}
+    M, N, K = args["M"], args["N"], args["K"]
+    ret["name"] = f"{kernel.name} [M={M}, N={N}, K={K}]"
+    if "c_ptr" in args:
+        bytes_per_elem = args["c_ptr"].element_size()
+    elif "c_desc_ptr" in args:
+        bytes_per_elem = 2
+    else:
+        bytes_per_elem = 1 if args["FP8_OUTPUT"] else 2
+    ret[f"flops{bytes_per_elem * 8}"] = 2. * M * N * K
+    ret["bytes"] = bytes_per_elem * (M * K + N * K + M * N)
+    return ret
+
+
+def matmul_get_configs():
+    return [
+        triton.Config({'BLOCK_SIZE_M': BM, 'BLOCK_SIZE_N': BN, "BLOCK_SIZE_K" : BK, "USE_TMA" : USE_TMA}, num_stages=s, num_warps=w) \
+        for BM in [128] \
+        for BN in [128, 256] \
+        for BK in [64, 128] \
+        for USE_TMA in [True, False] \
+        for s in ([3, 4]) \
+        for w in [4, 8] \
+    ]
+
+
+@triton.autotune(
+    configs=matmul_get_configs(),
+    key=["M", "N", "K"],
+)
+@triton.jit(launch_metadata=_matmul_launch_metadata)
+def matmul_kernel(a_ptr, b_ptr, c_ptr,  #
+                  M, N, K,  #
+                  BLOCK_SIZE_M: tl.constexpr,  #
+                  BLOCK_SIZE_N: tl.constexpr,  #
+                  BLOCK_SIZE_K: tl.constexpr,  #
+                  USE_TMA: tl.constexpr,  #
+                  USE_GDC: tl.constexpr):  #
+    # This kernel can use TMA or traditional tl.load/tl.store
+    a_desc = None
+    b_desc = None
+    c_desc = None
+    a_ptrs = None
+    b_ptrs = None
+    c_ptrs = None
+
+    if USE_TMA:
+        a_desc = tl.make_tensor_descriptor(
+            a_ptr,
+            shape=[M, K],
+            strides=[K, 1],
+            block_shape=[BLOCK_SIZE_M, BLOCK_SIZE_K],
+        )
+        b_desc = tl.make_tensor_descriptor(
+            b_ptr,
+            shape=[N, K],
+            strides=[K, 1],
+            block_shape=[BLOCK_SIZE_N, BLOCK_SIZE_K],
+        )
+        c_desc = tl.make_tensor_descriptor(
+            c_ptr,
+            shape=[M, N],
+            strides=[N, 1],
+            block_shape=[BLOCK_SIZE_M, BLOCK_SIZE_N],
+        )
+
+    dtype = c_ptr.dtype.element_ty
+    start_pid = tl.program_id(axis=0)
+    num_pid_m = tl.cdiv(M, BLOCK_SIZE_M)
+    k_tiles = tl.cdiv(K, BLOCK_SIZE_K)
+
+    pid_m = start_pid % num_pid_m
+    pid_n = start_pid // num_pid_m
+
+    if USE_TMA:
+        offs_am = pid_m * BLOCK_SIZE_M
+        offs_bn = pid_n * BLOCK_SIZE_N
+    else:
+        offs_am = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+        offs_bn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+        offs_am = tl.where(offs_am < M, offs_am, 0)
+        offs_bn = tl.where(offs_bn < N, offs_bn, 0)
+        offs_am = tl.max_contiguous(tl.multiple_of(offs_am, BLOCK_SIZE_M), BLOCK_SIZE_M)
+        offs_bn = tl.max_contiguous(tl.multiple_of(offs_bn, BLOCK_SIZE_N), BLOCK_SIZE_N)
+        offs_k = tl.arange(0, BLOCK_SIZE_K)
+        a_ptrs = a_ptr + (offs_am[:, None] * K + offs_k[None, :])
+        # B is N x K
+        b_ptrs = b_ptr + (offs_k[None, :] + offs_bn[:, None] * K)
+
+    accumulator = tl.zeros((BLOCK_SIZE_M, BLOCK_SIZE_N), dtype=tl.float32)
+
+    if USE_GDC:
+        # GDC wait is used to wait for the prior kernel to complete before continuing.
+        # If we utilize Programatic Dependent Launch, we must wait on the prior kernel
+        # to complete in case a or b are written to by the prior kernel.
+        # This is done to prevent races.
+        tl.extra.cuda.gdc_wait()
+
+    for ki in range(k_tiles):
+        a = None
+        b = None
+        if USE_TMA:
+            offs_k = ki * BLOCK_SIZE_K
+            a = a_desc.load([offs_am, offs_k])
+            b = b_desc.load([offs_bn, offs_k])
+        else:
+            a = tl.load(a_ptrs, mask=offs_k[None, :] < K - ki * BLOCK_SIZE_K, other=0.0)
+            b = tl.load(b_ptrs, mask=offs_k[None, :] < K - ki * BLOCK_SIZE_K, other=0.0)
+            a_ptrs += BLOCK_SIZE_K
+            b_ptrs += BLOCK_SIZE_K
+        accumulator = tl.dot(a, b.T, accumulator)
+
+    if USE_GDC:
+        # GDC launch dependents is used to launch dependent kernels.
+        # Once GDC launch it is possible for the next kernel to begin if
+        # there are enough resources.
+        tl.extra.cuda.gdc_launch_dependents()
+
+    c = accumulator.to(dtype)
+    if USE_TMA:
+        c_desc.store([offs_am, offs_bn], c)
+    else:
+        offs_cm = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+        offs_cn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+        c_ptrs = c_ptr + N * offs_cm[:, None] + offs_cn[None, :]
+        c_mask = (offs_cm[:, None] < M) & (offs_cn[None, :] < N)
+        tl.store(c_ptrs, c, mask=c_mask)
+
+
+def matmul(a, b, launch_pdl=True):
+    # Check constraints.
+    assert a.shape[1] == b.shape[1], "Incompatible dimensions"  # b is transposed
+    assert a.dtype == b.dtype, "Incompatible dtypes"
+
+    M, K = a.shape
+    N, K = b.shape
+    dtype = a.dtype
+
+    c = torch.empty((M, N), device=a.device, dtype=dtype)
+
+    # TMA descriptors require a global memory allocation
+    def alloc_fn(size: int, alignment: int, stream: Optional[int]):
+        return torch.empty(size, device="cuda", dtype=torch.int8)
path: python/tutorials/11-programmatic-dependent-launch.py
user: login: peterbell10
body: I think there is a potential race where the pytorch allocator may return the same underlying memory regions for the workspace allocations for two kernels, expecting that the stream means they don't collide. Then since you write the TMA descriptors before calling `gdc_wait` that means the previous kernel may not be finished with the workspace memory while the next kernel is overwriting it.
created_at: 2025-04-03T22:53:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,262 @@
+"""
+Programmatic Dependent Launch
+=====================
+This script demonstrates the use of programmatic dependent launch ontop of a non-persistent implementation of matrix multiplication using Triton.
+The kernels support both FP16 and FP8 data types but the FP8 implementation is only available on CUDA devices with compute capability >= 9.0.
+
+.. code-block:: bash
+    python 11-programmatic-dependent-launch.py
+"""
+
+import torch
+import triton
+import triton.language as tl
+
+from typing import Optional
+
+
+def is_cuda():
+    return triton.runtime.driver.active.get_current_target().backend == "cuda"
+
+
+def supports_tma():
+    return is_cuda() and torch.cuda.get_device_capability()[0] >= 9
+
+
+def _matmul_launch_metadata(grid, kernel, args):
+    ret = {}
+    M, N, K = args["M"], args["N"], args["K"]
+    ret["name"] = f"{kernel.name} [M={M}, N={N}, K={K}]"
+    if "c_ptr" in args:
+        bytes_per_elem = args["c_ptr"].element_size()
+    elif "c_desc_ptr" in args:
+        bytes_per_elem = 2
+    else:
+        bytes_per_elem = 1 if args["FP8_OUTPUT"] else 2
+    ret[f"flops{bytes_per_elem * 8}"] = 2. * M * N * K
+    ret["bytes"] = bytes_per_elem * (M * K + N * K + M * N)
+    return ret
+
+
+def matmul_get_configs():
+    return [
+        triton.Config({'BLOCK_SIZE_M': BM, 'BLOCK_SIZE_N': BN, "BLOCK_SIZE_K" : BK, "USE_TMA" : USE_TMA}, num_stages=s, num_warps=w) \
+        for BM in [128] \
+        for BN in [128, 256] \
+        for BK in [64, 128] \
+        for USE_TMA in [True, False] \
+        for s in ([3, 4]) \
+        for w in [4, 8] \
+    ]
+
+
+@triton.autotune(
+    configs=matmul_get_configs(),
+    key=["M", "N", "K"],
+)
+@triton.jit(launch_metadata=_matmul_launch_metadata)
+def matmul_kernel(a_ptr, b_ptr, c_ptr,  #
+                  M, N, K,  #
+                  BLOCK_SIZE_M: tl.constexpr,  #
+                  BLOCK_SIZE_N: tl.constexpr,  #
+                  BLOCK_SIZE_K: tl.constexpr,  #
+                  USE_TMA: tl.constexpr,  #
+                  USE_GDC: tl.constexpr):  #
+    # This kernel can use TMA or traditional tl.load/tl.store
+    a_desc = None
+    b_desc = None
+    c_desc = None
+    a_ptrs = None
+    b_ptrs = None
+    c_ptrs = None
+
+    if USE_TMA:
+        a_desc = tl.make_tensor_descriptor(
+            a_ptr,
+            shape=[M, K],
+            strides=[K, 1],
+            block_shape=[BLOCK_SIZE_M, BLOCK_SIZE_K],
+        )
+        b_desc = tl.make_tensor_descriptor(
+            b_ptr,
+            shape=[N, K],
+            strides=[K, 1],
+            block_shape=[BLOCK_SIZE_N, BLOCK_SIZE_K],
+        )
+        c_desc = tl.make_tensor_descriptor(
+            c_ptr,
+            shape=[M, N],
+            strides=[N, 1],
+            block_shape=[BLOCK_SIZE_M, BLOCK_SIZE_N],
+        )
+
+    dtype = c_ptr.dtype.element_ty
+    start_pid = tl.program_id(axis=0)
+    num_pid_m = tl.cdiv(M, BLOCK_SIZE_M)
+    k_tiles = tl.cdiv(K, BLOCK_SIZE_K)
+
+    pid_m = start_pid % num_pid_m
+    pid_n = start_pid // num_pid_m
+
+    if USE_TMA:
+        offs_am = pid_m * BLOCK_SIZE_M
+        offs_bn = pid_n * BLOCK_SIZE_N
+    else:
+        offs_am = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+        offs_bn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+        offs_am = tl.where(offs_am < M, offs_am, 0)
+        offs_bn = tl.where(offs_bn < N, offs_bn, 0)
+        offs_am = tl.max_contiguous(tl.multiple_of(offs_am, BLOCK_SIZE_M), BLOCK_SIZE_M)
+        offs_bn = tl.max_contiguous(tl.multiple_of(offs_bn, BLOCK_SIZE_N), BLOCK_SIZE_N)
+        offs_k = tl.arange(0, BLOCK_SIZE_K)
+        a_ptrs = a_ptr + (offs_am[:, None] * K + offs_k[None, :])
+        # B is N x K
+        b_ptrs = b_ptr + (offs_k[None, :] + offs_bn[:, None] * K)
+
+    accumulator = tl.zeros((BLOCK_SIZE_M, BLOCK_SIZE_N), dtype=tl.float32)
+
+    if USE_GDC:
+        # GDC wait is used to wait for the prior kernel to complete before continuing.
+        # If we utilize Programatic Dependent Launch, we must wait on the prior kernel
+        # to complete in case a or b are written to by the prior kernel.
+        # This is done to prevent races.
+        tl.extra.cuda.gdc_wait()
+
+    for ki in range(k_tiles):
+        a = None
+        b = None
+        if USE_TMA:
+            offs_k = ki * BLOCK_SIZE_K
+            a = a_desc.load([offs_am, offs_k])
+            b = b_desc.load([offs_bn, offs_k])
+        else:
+            a = tl.load(a_ptrs, mask=offs_k[None, :] < K - ki * BLOCK_SIZE_K, other=0.0)
+            b = tl.load(b_ptrs, mask=offs_k[None, :] < K - ki * BLOCK_SIZE_K, other=0.0)
+            a_ptrs += BLOCK_SIZE_K
+            b_ptrs += BLOCK_SIZE_K
+        accumulator = tl.dot(a, b.T, accumulator)
+
+    if USE_GDC:
+        # GDC launch dependents is used to launch dependent kernels.
+        # Once GDC launch it is possible for the next kernel to begin if
+        # there are enough resources.
+        tl.extra.cuda.gdc_launch_dependents()
+
+    c = accumulator.to(dtype)
+    if USE_TMA:
+        c_desc.store([offs_am, offs_bn], c)
+    else:
+        offs_cm = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+        offs_cn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+        c_ptrs = c_ptr + N * offs_cm[:, None] + offs_cn[None, :]
+        c_mask = (offs_cm[:, None] < M) & (offs_cn[None, :] < N)
+        tl.store(c_ptrs, c, mask=c_mask)
+
+
+def matmul(a, b, launch_pdl=True):
+    # Check constraints.
+    assert a.shape[1] == b.shape[1], "Incompatible dimensions"  # b is transposed
+    assert a.dtype == b.dtype, "Incompatible dtypes"
+
+    M, K = a.shape
+    N, K = b.shape
+    dtype = a.dtype
+
+    c = torch.empty((M, N), device=a.device, dtype=dtype)
+
+    # TMA descriptors require a global memory allocation
+    def alloc_fn(size: int, alignment: int, stream: Optional[int]):
+        return torch.empty(size, device="cuda", dtype=torch.int8)
path: python/tutorials/11-programmatic-dependent-launch.py
user: login: depaulmillz
body: Thanks for catching. I updated the kernel to make no assumptions about what the alloc fn may do. Performance is similar to before.
![image](https://github.com/user-attachments/assets/ef31288a-8ddd-45d8-ad94-c98ed7a76bec)

created_at: 2025-04-07T21:48:53Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,13 @@
+from triton.language import core
+
+
+@core.extern
+def gdc_wait(_builder=None):
+    return core.inline_asm_elementwise("griddepcontrol.wait; // dummy $0", "=r", [], dtype=core.int32, is_pure=False,
+                                       pack=1, _builder=_builder)
+
+
+@core.extern
+def gdc_launch_dependents(_builder=None):
+    return core.inline_asm_elementwise("griddepcontrol.launch_dependents; // dummy $0", "=r", [], dtype=core.int32,
+                                       is_pure=True, pack=1, _builder=_builder)
path: third_party/nvidia/language/cuda/gdc.py
user: login: peterbell10
body: Why do you say this is pure? Shouldn't it be DCE'd if it's pure.
created_at: 2025-04-09T20:07:27Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,265 @@
+"""
+Programmatic Dependent Launch
+=====================
+This script demonstrates the use of programmatic dependent launch ontop of a non-persistent implementation of matrix multiplication using Triton.
+The kernels support both FP16 and FP8 data types but the FP8 implementation is only available on CUDA devices with compute capability >= 9.0.
+
+.. code-block:: bash
+    python 11-programmatic-dependent-launch.py
+"""
+
+import torch
+import triton
+import triton.language as tl
+
+from typing import Optional
+
+
+def is_cuda():
+    return triton.runtime.driver.active.get_current_target().backend == "cuda"
+
+
+def supports_tma():
+    return is_cuda() and torch.cuda.get_device_capability()[0] >= 9
+
+
+def _matmul_launch_metadata(grid, kernel, args):
+    ret = {}
+    M, N, K = args["M"], args["N"], args["K"]
+    ret["name"] = f"{kernel.name} [M={M}, N={N}, K={K}]"
+    if "c_ptr" in args:
+        bytes_per_elem = args["c_ptr"].element_size()
+    elif "c_desc_ptr" in args:
+        bytes_per_elem = 2
+    else:
+        bytes_per_elem = 1 if args["FP8_OUTPUT"] else 2
+    ret[f"flops{bytes_per_elem * 8}"] = 2. * M * N * K
+    ret["bytes"] = bytes_per_elem * (M * K + N * K + M * N)
+    return ret
+
+
+def matmul_get_configs():
+    return [
+        triton.Config({'BLOCK_SIZE_M': BM, 'BLOCK_SIZE_N': BN, "BLOCK_SIZE_K" : BK, "USE_TMA" : USE_TMA}, num_stages=s, num_warps=w) \
+        for BM in [128] \
+        for BN in [128, 256] \
+        for BK in [64, 128] \
+        for USE_TMA in [True, False] \
+        for s in ([3, 4]) \
+        for w in [4, 8] \
+    ]
+
+
+@triton.autotune(
+    configs=matmul_get_configs(),
+    key=["M", "N", "K"],
+)
+@triton.jit(launch_metadata=_matmul_launch_metadata)
+def matmul_kernel(a_ptr, b_ptr, c_ptr,  #
+                  M, N, K,  #
+                  BLOCK_SIZE_M: tl.constexpr,  #
+                  BLOCK_SIZE_N: tl.constexpr,  #
+                  BLOCK_SIZE_K: tl.constexpr,  #
+                  USE_TMA: tl.constexpr,  #
+                  USE_GDC: tl.constexpr):  #
+    # This kernel can use TMA or traditional tl.load/tl.store
+    a_desc = None
+    b_desc = None
+    c_desc = None
+    a_ptrs = None
+    b_ptrs = None
+    c_ptrs = None
+
+    dtype = c_ptr.dtype.element_ty
+    start_pid = tl.program_id(axis=0)
+    num_pid_m = tl.cdiv(M, BLOCK_SIZE_M)
+    k_tiles = tl.cdiv(K, BLOCK_SIZE_K)
+
+    pid_m = start_pid % num_pid_m
+    pid_n = start_pid // num_pid_m
+
+    if USE_TMA:
+        offs_am = pid_m * BLOCK_SIZE_M
+        offs_bn = pid_n * BLOCK_SIZE_N
+    else:
+        offs_am = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+        offs_bn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+        offs_am = tl.where(offs_am < M, offs_am, 0)
+        offs_bn = tl.where(offs_bn < N, offs_bn, 0)
+        offs_am = tl.max_contiguous(tl.multiple_of(offs_am, BLOCK_SIZE_M), BLOCK_SIZE_M)
+        offs_bn = tl.max_contiguous(tl.multiple_of(offs_bn, BLOCK_SIZE_N), BLOCK_SIZE_N)
+        offs_k = tl.arange(0, BLOCK_SIZE_K)
+        a_ptrs = a_ptr + (offs_am[:, None] * K + offs_k[None, :])
+        # B is N x K
+        b_ptrs = b_ptr + (offs_k[None, :] + offs_bn[:, None] * K)
+
+    accumulator = tl.zeros((BLOCK_SIZE_M, BLOCK_SIZE_N), dtype=tl.float32)
+
+    if USE_GDC:
+        # GDC wait is used to wait for the prior kernel to complete before continuing.
+        # If we utilize Programmatic Dependent Launch, we must wait on the prior kernel
+        # to complete in case a or b are written to by the prior kernel.
+        # This is done to prevent races.
+        tl.extra.cuda.gdc_wait()
+
+    if USE_TMA:
+        # We have no guarentees about what memory may be passed for our make tensor descriptor,
+        # and whether there is inter kernel dependencies. We therefore must safely allocate after
+        # Waiting on the prior kernel to finish.
+        a_desc = tl.make_tensor_descriptor(
+            a_ptr,
+            shape=[M, K],
+            strides=[K, 1],
+            block_shape=[BLOCK_SIZE_M, BLOCK_SIZE_K],
+        )
+        b_desc = tl.make_tensor_descriptor(
+            b_ptr,
+            shape=[N, K],
+            strides=[K, 1],
+            block_shape=[BLOCK_SIZE_N, BLOCK_SIZE_K],
+        )
+        c_desc = tl.make_tensor_descriptor(
+            c_ptr,
+            shape=[M, N],
+            strides=[N, 1],
+            block_shape=[BLOCK_SIZE_M, BLOCK_SIZE_N],
+        )
+
+    for ki in range(k_tiles):
+        a = None
+        b = None
+        if USE_TMA:
+            offs_k = ki * BLOCK_SIZE_K
+            a = a_desc.load([offs_am, offs_k])
+            b = b_desc.load([offs_bn, offs_k])
+        else:
+            a = tl.load(a_ptrs, mask=offs_k[None, :] < K - ki * BLOCK_SIZE_K, other=0.0)
+            b = tl.load(b_ptrs, mask=offs_k[None, :] < K - ki * BLOCK_SIZE_K, other=0.0)
+            a_ptrs += BLOCK_SIZE_K
+            b_ptrs += BLOCK_SIZE_K
+        accumulator = tl.dot(a, b.T, accumulator)
+
+    if USE_GDC:
+        # GDC launch dependents is used to launch dependent kernels.
+        # Once GDC launch it is possible for the next kernel to begin if
+        # there are enough resources.
+        tl.extra.cuda.gdc_launch_dependents()
+
+    c = accumulator.to(dtype)
+    if USE_TMA:
+        c_desc.store([offs_am, offs_bn], c)
+    else:
+        offs_cm = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+        offs_cn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+        c_ptrs = c_ptr + N * offs_cm[:, None] + offs_cn[None, :]
+        c_mask = (offs_cm[:, None] < M) & (offs_cn[None, :] < N)
+        tl.store(c_ptrs, c, mask=c_mask)
+
+
+def matmul(a, b, launch_pdl=True):
+    # Check constraints.
+    assert a.shape[1] == b.shape[1], "Incompatible dimensions"  # b is transposed
+    assert a.dtype == b.dtype, "Incompatible dtypes"
+
+    M, K = a.shape
+    N, K = b.shape
+    dtype = a.dtype
+
+    c = torch.empty((M, N), device=a.device, dtype=dtype)
+
+    # TMA descriptors require a global memory allocation
+    def alloc_fn(size: int, alignment: int, stream: Optional[int]):
+        return torch.empty(size, device="cuda", dtype=torch.int8)
+
+    triton.set_allocator(alloc_fn)
+
+    grid = lambda META: (triton.cdiv(M, META["BLOCK_SIZE_M"]) * triton.cdiv(N, META["BLOCK_SIZE_N"]), )
+    matmul_kernel[grid](
path: python/tutorials/11-programmatic-dependent-launch.py
user: login: peterbell10
body: How do the benchmarks change if we use a persistent kernel? It seems kind of pointless to say we can get a huge speedup when you use a really inefficient kernel to begin with.
created_at: 2025-04-09T20:09:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,265 @@
+"""
+Programmatic Dependent Launch
+=====================
+This script demonstrates the use of programmatic dependent launch ontop of a non-persistent implementation of matrix multiplication using Triton.
+The kernels support both FP16 and FP8 data types but the FP8 implementation is only available on CUDA devices with compute capability >= 9.0.
+
+.. code-block:: bash
+    python 11-programmatic-dependent-launch.py
+"""
+
+import torch
+import triton
+import triton.language as tl
+
+from typing import Optional
+
+
+def is_cuda():
+    return triton.runtime.driver.active.get_current_target().backend == "cuda"
+
+
+def supports_tma():
+    return is_cuda() and torch.cuda.get_device_capability()[0] >= 9
+
+
+def _matmul_launch_metadata(grid, kernel, args):
+    ret = {}
+    M, N, K = args["M"], args["N"], args["K"]
+    ret["name"] = f"{kernel.name} [M={M}, N={N}, K={K}]"
+    if "c_ptr" in args:
+        bytes_per_elem = args["c_ptr"].element_size()
+    elif "c_desc_ptr" in args:
+        bytes_per_elem = 2
+    else:
+        bytes_per_elem = 1 if args["FP8_OUTPUT"] else 2
+    ret[f"flops{bytes_per_elem * 8}"] = 2. * M * N * K
+    ret["bytes"] = bytes_per_elem * (M * K + N * K + M * N)
+    return ret
+
+
+def matmul_get_configs():
+    return [
+        triton.Config({'BLOCK_SIZE_M': BM, 'BLOCK_SIZE_N': BN, "BLOCK_SIZE_K" : BK, "USE_TMA" : USE_TMA}, num_stages=s, num_warps=w) \
+        for BM in [128] \
+        for BN in [128, 256] \
+        for BK in [64, 128] \
+        for USE_TMA in [True, False] \
+        for s in ([3, 4]) \
+        for w in [4, 8] \
+    ]
+
+
+@triton.autotune(
+    configs=matmul_get_configs(),
+    key=["M", "N", "K"],
+)
+@triton.jit(launch_metadata=_matmul_launch_metadata)
+def matmul_kernel(a_ptr, b_ptr, c_ptr,  #
+                  M, N, K,  #
+                  BLOCK_SIZE_M: tl.constexpr,  #
+                  BLOCK_SIZE_N: tl.constexpr,  #
+                  BLOCK_SIZE_K: tl.constexpr,  #
+                  USE_TMA: tl.constexpr,  #
+                  USE_GDC: tl.constexpr):  #
+    # This kernel can use TMA or traditional tl.load/tl.store
+    a_desc = None
+    b_desc = None
+    c_desc = None
+    a_ptrs = None
+    b_ptrs = None
+    c_ptrs = None
+
+    dtype = c_ptr.dtype.element_ty
+    start_pid = tl.program_id(axis=0)
+    num_pid_m = tl.cdiv(M, BLOCK_SIZE_M)
+    k_tiles = tl.cdiv(K, BLOCK_SIZE_K)
+
+    pid_m = start_pid % num_pid_m
+    pid_n = start_pid // num_pid_m
+
+    if USE_TMA:
+        offs_am = pid_m * BLOCK_SIZE_M
+        offs_bn = pid_n * BLOCK_SIZE_N
+    else:
+        offs_am = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+        offs_bn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+        offs_am = tl.where(offs_am < M, offs_am, 0)
+        offs_bn = tl.where(offs_bn < N, offs_bn, 0)
+        offs_am = tl.max_contiguous(tl.multiple_of(offs_am, BLOCK_SIZE_M), BLOCK_SIZE_M)
+        offs_bn = tl.max_contiguous(tl.multiple_of(offs_bn, BLOCK_SIZE_N), BLOCK_SIZE_N)
+        offs_k = tl.arange(0, BLOCK_SIZE_K)
+        a_ptrs = a_ptr + (offs_am[:, None] * K + offs_k[None, :])
+        # B is N x K
+        b_ptrs = b_ptr + (offs_k[None, :] + offs_bn[:, None] * K)
+
+    accumulator = tl.zeros((BLOCK_SIZE_M, BLOCK_SIZE_N), dtype=tl.float32)
+
+    if USE_GDC:
+        # GDC wait is used to wait for the prior kernel to complete before continuing.
+        # If we utilize Programmatic Dependent Launch, we must wait on the prior kernel
+        # to complete in case a or b are written to by the prior kernel.
+        # This is done to prevent races.
+        tl.extra.cuda.gdc_wait()
+
+    if USE_TMA:
+        # We have no guarentees about what memory may be passed for our make tensor descriptor,
+        # and whether there is inter kernel dependencies. We therefore must safely allocate after
+        # Waiting on the prior kernel to finish.
+        a_desc = tl.make_tensor_descriptor(
+            a_ptr,
+            shape=[M, K],
+            strides=[K, 1],
+            block_shape=[BLOCK_SIZE_M, BLOCK_SIZE_K],
+        )
+        b_desc = tl.make_tensor_descriptor(
+            b_ptr,
+            shape=[N, K],
+            strides=[K, 1],
+            block_shape=[BLOCK_SIZE_N, BLOCK_SIZE_K],
+        )
+        c_desc = tl.make_tensor_descriptor(
+            c_ptr,
+            shape=[M, N],
+            strides=[N, 1],
+            block_shape=[BLOCK_SIZE_M, BLOCK_SIZE_N],
+        )
+
+    for ki in range(k_tiles):
+        a = None
+        b = None
+        if USE_TMA:
+            offs_k = ki * BLOCK_SIZE_K
+            a = a_desc.load([offs_am, offs_k])
+            b = b_desc.load([offs_bn, offs_k])
+        else:
+            a = tl.load(a_ptrs, mask=offs_k[None, :] < K - ki * BLOCK_SIZE_K, other=0.0)
+            b = tl.load(b_ptrs, mask=offs_k[None, :] < K - ki * BLOCK_SIZE_K, other=0.0)
+            a_ptrs += BLOCK_SIZE_K
+            b_ptrs += BLOCK_SIZE_K
+        accumulator = tl.dot(a, b.T, accumulator)
+
+    if USE_GDC:
+        # GDC launch dependents is used to launch dependent kernels.
+        # Once GDC launch it is possible for the next kernel to begin if
+        # there are enough resources.
+        tl.extra.cuda.gdc_launch_dependents()
+
+    c = accumulator.to(dtype)
+    if USE_TMA:
+        c_desc.store([offs_am, offs_bn], c)
+    else:
+        offs_cm = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+        offs_cn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+        c_ptrs = c_ptr + N * offs_cm[:, None] + offs_cn[None, :]
+        c_mask = (offs_cm[:, None] < M) & (offs_cn[None, :] < N)
+        tl.store(c_ptrs, c, mask=c_mask)
+
+
+def matmul(a, b, launch_pdl=True):
+    # Check constraints.
+    assert a.shape[1] == b.shape[1], "Incompatible dimensions"  # b is transposed
+    assert a.dtype == b.dtype, "Incompatible dtypes"
+
+    M, K = a.shape
+    N, K = b.shape
+    dtype = a.dtype
+
+    c = torch.empty((M, N), device=a.device, dtype=dtype)
+
+    # TMA descriptors require a global memory allocation
+    def alloc_fn(size: int, alignment: int, stream: Optional[int]):
+        return torch.empty(size, device="cuda", dtype=torch.int8)
+
+    triton.set_allocator(alloc_fn)
+
+    grid = lambda META: (triton.cdiv(M, META["BLOCK_SIZE_M"]) * triton.cdiv(N, META["BLOCK_SIZE_N"]), )
+    matmul_kernel[grid](
+        a, b, c,  #
+        M, N, K,  #
+        USE_GDC=launch_pdl,  # set constexpr in kernel to use grid dependence control
+        launch_pdl=launch_pdl,  # launch kernel with PDL flag set enabled
+    )
+    return c
+
+
+def validate(M, N, K, dtype):
+    a = torch.randn((M, K), device="cuda", dtype=torch.float16).to(dtype)
+    b = torch.randn((K, N), device="cuda", dtype=torch.float16).to(dtype)
+    b = b.T.contiguous()
+
+    torch_result = torch.matmul(a, b.T)
+    matmul_result = matmul(a, b)
+
+    torch_vs_matmul = "✅" if torch.allclose(torch_result.to(torch.float16), matmul_result.to(torch.float16),
+                                            atol=1.0) else "❌"
+    print(f"M={M}, N={N}, K={K} verification naive vs: ", end="")
+    print(f"matmul: {torch_vs_matmul} ", end="")
+    print()
+
+
+TORCH_HAS_FP8 = hasattr(torch, 'float8_e5m2')
+
+
+@triton.testing.perf_report(
+    triton.testing.Benchmark(
+        x_names=["K"],
+        x_vals=range(512, 4096 + 1, 512),
+        x_log=False,
+        line_arg="provider",
+        line_vals=["pdl-fp16", "fp16"] + (["pdl-fp8", "fp8"] if TORCH_HAS_FP8 else []),
+        line_names=["PDL [FP16]", "No PDL [FP16]"] + (["PDL [FP8]", "No PDL [FP8]"] if TORCH_HAS_FP8 else []),
+        styles=[("red", "-"), ("blue", "-"), ("green", "-"), ("orange", "-")],
+        ylabel="TFLOPS",
+        plot_name="pdl-performance",
+        args={},
+    ))
+def benchmark(K, provider, M, N):
+
+    tflops = (lambda ms: (2.0 * M * N * K) * 1e-12 / (ms * 1e-3))
+
+    quantiles = [0.5, 0.2, 0.8]
+    if "fp8" in provider:
+        a = torch.randn((M, K), device="cuda", dtype=torch.float16).to(torch.float8_e5m2)
+        b = torch.randn((K, N), device="cuda", dtype=torch.float16).to(torch.float8_e5m2)
+        b = b.T.contiguous()
+        if "pdl" in provider:
+            ms, min_ms, max_ms = triton.testing.do_bench_cudagraph(
+                lambda: matmul(a, b),
+                quantiles=quantiles,
+                rep=1000,
+            )
+            return tflops(ms), tflops(max_ms), tflops(min_ms)
+        else:
+            ms, min_ms, max_ms = triton.testing.do_bench_cudagraph(
+                lambda: matmul(a, b, False),
+                quantiles=quantiles,
+                rep=1000,
+            )
+            return tflops(ms), tflops(max_ms), tflops(min_ms)
+    else:
+        a = torch.randn((M, K), device="cuda", dtype=torch.float16)
+        b = torch.randn((K, N), device="cuda", dtype=torch.float16)
+        b = b.T.contiguous()
+        if "pdl" in provider:
+            ms, min_ms, max_ms = triton.testing.do_bench_cudagraph(
+                lambda: matmul(a, b),
+                quantiles=quantiles,
+                rep=1000,
+            )
+            return tflops(ms), tflops(max_ms), tflops(min_ms)
+        else:
+            ms, min_ms, max_ms = triton.testing.do_bench_cudagraph(
+                lambda: matmul(a, b, False),
+                quantiles=quantiles,
+                rep=1000,
+            )
+            return tflops(ms), tflops(max_ms), tflops(min_ms)
path: python/tutorials/11-programmatic-dependent-launch.py
user: login: peterbell10
body: NIT: This is unnecessarily verbose
```suggestion
    dtype = torch.float8_e5m2 if "fp8" in provider else torch.float16
    a = torch.randn((M, K), device="cuda", dtype=torch.float16).to(dtype)
    b = torch.randn((N, K), device="cuda", dtype=torch.float16).to(dtype)
    
    use_pdl = "pdl" in provider
    ms, min_ms, max_ms = triton.testing.do_bench_cudagraph(
        lambda: matmul(a, b, use_pdl),
        quantiles=quantiles,
        rep=1000,
    )
    return tflops(ms), tflops(max_ms), tflops(min_ms)
```
created_at: 2025-04-09T20:15:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,265 @@
+"""
+Programmatic Dependent Launch
+=====================
+This script demonstrates the use of programmatic dependent launch ontop of a non-persistent implementation of matrix multiplication using Triton.
+The kernels support both FP16 and FP8 data types but the FP8 implementation is only available on CUDA devices with compute capability >= 9.0.
+
+.. code-block:: bash
+    python 11-programmatic-dependent-launch.py
+"""
+
+import torch
+import triton
+import triton.language as tl
+
+from typing import Optional
+
+
+def is_cuda():
+    return triton.runtime.driver.active.get_current_target().backend == "cuda"
+
+
+def supports_tma():
+    return is_cuda() and torch.cuda.get_device_capability()[0] >= 9
+
+
+def _matmul_launch_metadata(grid, kernel, args):
+    ret = {}
+    M, N, K = args["M"], args["N"], args["K"]
+    ret["name"] = f"{kernel.name} [M={M}, N={N}, K={K}]"
+    if "c_ptr" in args:
+        bytes_per_elem = args["c_ptr"].element_size()
+    elif "c_desc_ptr" in args:
+        bytes_per_elem = 2
+    else:
+        bytes_per_elem = 1 if args["FP8_OUTPUT"] else 2
+    ret[f"flops{bytes_per_elem * 8}"] = 2. * M * N * K
+    ret["bytes"] = bytes_per_elem * (M * K + N * K + M * N)
+    return ret
+
+
+def matmul_get_configs():
+    return [
+        triton.Config({'BLOCK_SIZE_M': BM, 'BLOCK_SIZE_N': BN, "BLOCK_SIZE_K" : BK, "USE_TMA" : USE_TMA}, num_stages=s, num_warps=w) \
+        for BM in [128] \
+        for BN in [128, 256] \
+        for BK in [64, 128] \
+        for USE_TMA in [True, False] \
+        for s in ([3, 4]) \
+        for w in [4, 8] \
+    ]
+
+
+@triton.autotune(
+    configs=matmul_get_configs(),
+    key=["M", "N", "K"],
+)
+@triton.jit(launch_metadata=_matmul_launch_metadata)
+def matmul_kernel(a_ptr, b_ptr, c_ptr,  #
+                  M, N, K,  #
+                  BLOCK_SIZE_M: tl.constexpr,  #
+                  BLOCK_SIZE_N: tl.constexpr,  #
+                  BLOCK_SIZE_K: tl.constexpr,  #
+                  USE_TMA: tl.constexpr,  #
+                  USE_GDC: tl.constexpr):  #
+    # This kernel can use TMA or traditional tl.load/tl.store
+    a_desc = None
+    b_desc = None
+    c_desc = None
+    a_ptrs = None
+    b_ptrs = None
+    c_ptrs = None
+
+    dtype = c_ptr.dtype.element_ty
+    start_pid = tl.program_id(axis=0)
+    num_pid_m = tl.cdiv(M, BLOCK_SIZE_M)
+    k_tiles = tl.cdiv(K, BLOCK_SIZE_K)
+
+    pid_m = start_pid % num_pid_m
+    pid_n = start_pid // num_pid_m
+
+    if USE_TMA:
+        offs_am = pid_m * BLOCK_SIZE_M
+        offs_bn = pid_n * BLOCK_SIZE_N
+    else:
+        offs_am = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+        offs_bn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+        offs_am = tl.where(offs_am < M, offs_am, 0)
+        offs_bn = tl.where(offs_bn < N, offs_bn, 0)
+        offs_am = tl.max_contiguous(tl.multiple_of(offs_am, BLOCK_SIZE_M), BLOCK_SIZE_M)
+        offs_bn = tl.max_contiguous(tl.multiple_of(offs_bn, BLOCK_SIZE_N), BLOCK_SIZE_N)
+        offs_k = tl.arange(0, BLOCK_SIZE_K)
+        a_ptrs = a_ptr + (offs_am[:, None] * K + offs_k[None, :])
+        # B is N x K
+        b_ptrs = b_ptr + (offs_k[None, :] + offs_bn[:, None] * K)
+
+    accumulator = tl.zeros((BLOCK_SIZE_M, BLOCK_SIZE_N), dtype=tl.float32)
+
+    if USE_GDC:
+        # GDC wait is used to wait for the prior kernel to complete before continuing.
+        # If we utilize Programmatic Dependent Launch, we must wait on the prior kernel
+        # to complete in case a or b are written to by the prior kernel.
+        # This is done to prevent races.
+        tl.extra.cuda.gdc_wait()
+
+    if USE_TMA:
+        # We have no guarentees about what memory may be passed for our make tensor descriptor,
+        # and whether there is inter kernel dependencies. We therefore must safely allocate after
+        # Waiting on the prior kernel to finish.
+        a_desc = tl.make_tensor_descriptor(
+            a_ptr,
+            shape=[M, K],
+            strides=[K, 1],
+            block_shape=[BLOCK_SIZE_M, BLOCK_SIZE_K],
+        )
+        b_desc = tl.make_tensor_descriptor(
+            b_ptr,
+            shape=[N, K],
+            strides=[K, 1],
+            block_shape=[BLOCK_SIZE_N, BLOCK_SIZE_K],
+        )
+        c_desc = tl.make_tensor_descriptor(
+            c_ptr,
+            shape=[M, N],
+            strides=[N, 1],
+            block_shape=[BLOCK_SIZE_M, BLOCK_SIZE_N],
+        )
+
+    for ki in range(k_tiles):
+        a = None
+        b = None
+        if USE_TMA:
+            offs_k = ki * BLOCK_SIZE_K
+            a = a_desc.load([offs_am, offs_k])
+            b = b_desc.load([offs_bn, offs_k])
+        else:
+            a = tl.load(a_ptrs, mask=offs_k[None, :] < K - ki * BLOCK_SIZE_K, other=0.0)
+            b = tl.load(b_ptrs, mask=offs_k[None, :] < K - ki * BLOCK_SIZE_K, other=0.0)
+            a_ptrs += BLOCK_SIZE_K
+            b_ptrs += BLOCK_SIZE_K
+        accumulator = tl.dot(a, b.T, accumulator)
+
+    if USE_GDC:
+        # GDC launch dependents is used to launch dependent kernels.
+        # Once GDC launch it is possible for the next kernel to begin if
+        # there are enough resources.
+        tl.extra.cuda.gdc_launch_dependents()
+
+    c = accumulator.to(dtype)
+    if USE_TMA:
+        c_desc.store([offs_am, offs_bn], c)
+    else:
+        offs_cm = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+        offs_cn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+        c_ptrs = c_ptr + N * offs_cm[:, None] + offs_cn[None, :]
+        c_mask = (offs_cm[:, None] < M) & (offs_cn[None, :] < N)
+        tl.store(c_ptrs, c, mask=c_mask)
+
+
+def matmul(a, b, launch_pdl=True):
+    # Check constraints.
+    assert a.shape[1] == b.shape[1], "Incompatible dimensions"  # b is transposed
+    assert a.dtype == b.dtype, "Incompatible dtypes"
+
+    M, K = a.shape
+    N, K = b.shape
+    dtype = a.dtype
+
+    c = torch.empty((M, N), device=a.device, dtype=dtype)
+
+    # TMA descriptors require a global memory allocation
+    def alloc_fn(size: int, alignment: int, stream: Optional[int]):
+        return torch.empty(size, device="cuda", dtype=torch.int8)
+
+    triton.set_allocator(alloc_fn)
+
+    grid = lambda META: (triton.cdiv(M, META["BLOCK_SIZE_M"]) * triton.cdiv(N, META["BLOCK_SIZE_N"]), )
+    matmul_kernel[grid](
path: python/tutorials/11-programmatic-dependent-launch.py
user: login: peterbell10
body: Hypothetically, it may still be a good speedup since the work partitioning is entirely static for a persistent matmul so there may be more idle SMs while one kernel ramps down. Would be interesting to see. 
created_at: 2025-04-09T20:26:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,265 @@
+"""
+Programmatic Dependent Launch
+=====================
+This script demonstrates the use of programmatic dependent launch ontop of a non-persistent implementation of matrix multiplication using Triton.
+The kernels support both FP16 and FP8 data types but the FP8 implementation is only available on CUDA devices with compute capability >= 9.0.
+
+.. code-block:: bash
+    python 11-programmatic-dependent-launch.py
+"""
+
+import torch
+import triton
+import triton.language as tl
+
+from typing import Optional
+
+
+def is_cuda():
+    return triton.runtime.driver.active.get_current_target().backend == "cuda"
+
+
+def supports_tma():
+    return is_cuda() and torch.cuda.get_device_capability()[0] >= 9
+
+
+def _matmul_launch_metadata(grid, kernel, args):
+    ret = {}
+    M, N, K = args["M"], args["N"], args["K"]
+    ret["name"] = f"{kernel.name} [M={M}, N={N}, K={K}]"
+    if "c_ptr" in args:
+        bytes_per_elem = args["c_ptr"].element_size()
+    elif "c_desc_ptr" in args:
+        bytes_per_elem = 2
+    else:
+        bytes_per_elem = 1 if args["FP8_OUTPUT"] else 2
+    ret[f"flops{bytes_per_elem * 8}"] = 2. * M * N * K
+    ret["bytes"] = bytes_per_elem * (M * K + N * K + M * N)
+    return ret
+
+
+def matmul_get_configs():
+    return [
+        triton.Config({'BLOCK_SIZE_M': BM, 'BLOCK_SIZE_N': BN, "BLOCK_SIZE_K" : BK, "USE_TMA" : USE_TMA}, num_stages=s, num_warps=w) \
+        for BM in [128] \
+        for BN in [128, 256] \
+        for BK in [64, 128] \
+        for USE_TMA in [True, False] \
+        for s in ([3, 4]) \
+        for w in [4, 8] \
+    ]
+
+
+@triton.autotune(
+    configs=matmul_get_configs(),
+    key=["M", "N", "K"],
+)
+@triton.jit(launch_metadata=_matmul_launch_metadata)
+def matmul_kernel(a_ptr, b_ptr, c_ptr,  #
+                  M, N, K,  #
+                  BLOCK_SIZE_M: tl.constexpr,  #
+                  BLOCK_SIZE_N: tl.constexpr,  #
+                  BLOCK_SIZE_K: tl.constexpr,  #
+                  USE_TMA: tl.constexpr,  #
+                  USE_GDC: tl.constexpr):  #
+    # This kernel can use TMA or traditional tl.load/tl.store
+    a_desc = None
+    b_desc = None
+    c_desc = None
+    a_ptrs = None
+    b_ptrs = None
+    c_ptrs = None
+
+    dtype = c_ptr.dtype.element_ty
+    start_pid = tl.program_id(axis=0)
+    num_pid_m = tl.cdiv(M, BLOCK_SIZE_M)
+    k_tiles = tl.cdiv(K, BLOCK_SIZE_K)
+
+    pid_m = start_pid % num_pid_m
+    pid_n = start_pid // num_pid_m
+
+    if USE_TMA:
+        offs_am = pid_m * BLOCK_SIZE_M
+        offs_bn = pid_n * BLOCK_SIZE_N
+    else:
+        offs_am = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+        offs_bn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+        offs_am = tl.where(offs_am < M, offs_am, 0)
+        offs_bn = tl.where(offs_bn < N, offs_bn, 0)
+        offs_am = tl.max_contiguous(tl.multiple_of(offs_am, BLOCK_SIZE_M), BLOCK_SIZE_M)
+        offs_bn = tl.max_contiguous(tl.multiple_of(offs_bn, BLOCK_SIZE_N), BLOCK_SIZE_N)
+        offs_k = tl.arange(0, BLOCK_SIZE_K)
+        a_ptrs = a_ptr + (offs_am[:, None] * K + offs_k[None, :])
+        # B is N x K
+        b_ptrs = b_ptr + (offs_k[None, :] + offs_bn[:, None] * K)
+
+    accumulator = tl.zeros((BLOCK_SIZE_M, BLOCK_SIZE_N), dtype=tl.float32)
+
+    if USE_GDC:
+        # GDC wait is used to wait for the prior kernel to complete before continuing.
+        # If we utilize Programmatic Dependent Launch, we must wait on the prior kernel
+        # to complete in case a or b are written to by the prior kernel.
+        # This is done to prevent races.
+        tl.extra.cuda.gdc_wait()
+
+    if USE_TMA:
+        # We have no guarentees about what memory may be passed for our make tensor descriptor,
+        # and whether there is inter kernel dependencies. We therefore must safely allocate after
+        # Waiting on the prior kernel to finish.
+        a_desc = tl.make_tensor_descriptor(
+            a_ptr,
+            shape=[M, K],
+            strides=[K, 1],
+            block_shape=[BLOCK_SIZE_M, BLOCK_SIZE_K],
+        )
+        b_desc = tl.make_tensor_descriptor(
+            b_ptr,
+            shape=[N, K],
+            strides=[K, 1],
+            block_shape=[BLOCK_SIZE_N, BLOCK_SIZE_K],
+        )
+        c_desc = tl.make_tensor_descriptor(
+            c_ptr,
+            shape=[M, N],
+            strides=[N, 1],
+            block_shape=[BLOCK_SIZE_M, BLOCK_SIZE_N],
+        )
+
+    for ki in range(k_tiles):
+        a = None
+        b = None
+        if USE_TMA:
+            offs_k = ki * BLOCK_SIZE_K
+            a = a_desc.load([offs_am, offs_k])
+            b = b_desc.load([offs_bn, offs_k])
+        else:
+            a = tl.load(a_ptrs, mask=offs_k[None, :] < K - ki * BLOCK_SIZE_K, other=0.0)
+            b = tl.load(b_ptrs, mask=offs_k[None, :] < K - ki * BLOCK_SIZE_K, other=0.0)
+            a_ptrs += BLOCK_SIZE_K
+            b_ptrs += BLOCK_SIZE_K
+        accumulator = tl.dot(a, b.T, accumulator)
+
+    if USE_GDC:
+        # GDC launch dependents is used to launch dependent kernels.
+        # Once GDC launch it is possible for the next kernel to begin if
+        # there are enough resources.
+        tl.extra.cuda.gdc_launch_dependents()
+
+    c = accumulator.to(dtype)
+    if USE_TMA:
+        c_desc.store([offs_am, offs_bn], c)
+    else:
+        offs_cm = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+        offs_cn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+        c_ptrs = c_ptr + N * offs_cm[:, None] + offs_cn[None, :]
+        c_mask = (offs_cm[:, None] < M) & (offs_cn[None, :] < N)
+        tl.store(c_ptrs, c, mask=c_mask)
+
+
+def matmul(a, b, launch_pdl=True):
+    # Check constraints.
+    assert a.shape[1] == b.shape[1], "Incompatible dimensions"  # b is transposed
+    assert a.dtype == b.dtype, "Incompatible dtypes"
+
+    M, K = a.shape
+    N, K = b.shape
+    dtype = a.dtype
+
+    c = torch.empty((M, N), device=a.device, dtype=dtype)
+
+    # TMA descriptors require a global memory allocation
+    def alloc_fn(size: int, alignment: int, stream: Optional[int]):
+        return torch.empty(size, device="cuda", dtype=torch.int8)
+
+    triton.set_allocator(alloc_fn)
+
+    grid = lambda META: (triton.cdiv(M, META["BLOCK_SIZE_M"]) * triton.cdiv(N, META["BLOCK_SIZE_N"]), )
+    matmul_kernel[grid](
path: python/tutorials/11-programmatic-dependent-launch.py
user: login: depaulmillz
body: The intent here was to focus on a single wave of computation. So, persistence does not matter in that case. I can add a persistent version as well.
created_at: 2025-04-09T20:37:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,265 @@
+"""
+Programmatic Dependent Launch
+=====================
+This script demonstrates the use of programmatic dependent launch ontop of a non-persistent implementation of matrix multiplication using Triton.
+The kernels support both FP16 and FP8 data types but the FP8 implementation is only available on CUDA devices with compute capability >= 9.0.
+
+.. code-block:: bash
+    python 11-programmatic-dependent-launch.py
+"""
+
+import torch
+import triton
+import triton.language as tl
+
+from typing import Optional
+
+
+def is_cuda():
+    return triton.runtime.driver.active.get_current_target().backend == "cuda"
+
+
+def supports_tma():
+    return is_cuda() and torch.cuda.get_device_capability()[0] >= 9
+
+
+def _matmul_launch_metadata(grid, kernel, args):
+    ret = {}
+    M, N, K = args["M"], args["N"], args["K"]
+    ret["name"] = f"{kernel.name} [M={M}, N={N}, K={K}]"
+    if "c_ptr" in args:
+        bytes_per_elem = args["c_ptr"].element_size()
+    elif "c_desc_ptr" in args:
+        bytes_per_elem = 2
+    else:
+        bytes_per_elem = 1 if args["FP8_OUTPUT"] else 2
+    ret[f"flops{bytes_per_elem * 8}"] = 2. * M * N * K
+    ret["bytes"] = bytes_per_elem * (M * K + N * K + M * N)
+    return ret
+
+
+def matmul_get_configs():
+    return [
+        triton.Config({'BLOCK_SIZE_M': BM, 'BLOCK_SIZE_N': BN, "BLOCK_SIZE_K" : BK, "USE_TMA" : USE_TMA}, num_stages=s, num_warps=w) \
+        for BM in [128] \
+        for BN in [128, 256] \
+        for BK in [64, 128] \
+        for USE_TMA in [True, False] \
+        for s in ([3, 4]) \
+        for w in [4, 8] \
+    ]
+
+
+@triton.autotune(
+    configs=matmul_get_configs(),
+    key=["M", "N", "K"],
+)
+@triton.jit(launch_metadata=_matmul_launch_metadata)
+def matmul_kernel(a_ptr, b_ptr, c_ptr,  #
+                  M, N, K,  #
+                  BLOCK_SIZE_M: tl.constexpr,  #
+                  BLOCK_SIZE_N: tl.constexpr,  #
+                  BLOCK_SIZE_K: tl.constexpr,  #
+                  USE_TMA: tl.constexpr,  #
+                  USE_GDC: tl.constexpr):  #
+    # This kernel can use TMA or traditional tl.load/tl.store
+    a_desc = None
+    b_desc = None
+    c_desc = None
+    a_ptrs = None
+    b_ptrs = None
+    c_ptrs = None
+
+    dtype = c_ptr.dtype.element_ty
+    start_pid = tl.program_id(axis=0)
+    num_pid_m = tl.cdiv(M, BLOCK_SIZE_M)
+    k_tiles = tl.cdiv(K, BLOCK_SIZE_K)
+
+    pid_m = start_pid % num_pid_m
+    pid_n = start_pid // num_pid_m
+
+    if USE_TMA:
+        offs_am = pid_m * BLOCK_SIZE_M
+        offs_bn = pid_n * BLOCK_SIZE_N
+    else:
+        offs_am = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+        offs_bn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+        offs_am = tl.where(offs_am < M, offs_am, 0)
+        offs_bn = tl.where(offs_bn < N, offs_bn, 0)
+        offs_am = tl.max_contiguous(tl.multiple_of(offs_am, BLOCK_SIZE_M), BLOCK_SIZE_M)
+        offs_bn = tl.max_contiguous(tl.multiple_of(offs_bn, BLOCK_SIZE_N), BLOCK_SIZE_N)
+        offs_k = tl.arange(0, BLOCK_SIZE_K)
+        a_ptrs = a_ptr + (offs_am[:, None] * K + offs_k[None, :])
+        # B is N x K
+        b_ptrs = b_ptr + (offs_k[None, :] + offs_bn[:, None] * K)
+
+    accumulator = tl.zeros((BLOCK_SIZE_M, BLOCK_SIZE_N), dtype=tl.float32)
+
+    if USE_GDC:
+        # GDC wait is used to wait for the prior kernel to complete before continuing.
+        # If we utilize Programmatic Dependent Launch, we must wait on the prior kernel
+        # to complete in case a or b are written to by the prior kernel.
+        # This is done to prevent races.
+        tl.extra.cuda.gdc_wait()
+
+    if USE_TMA:
+        # We have no guarentees about what memory may be passed for our make tensor descriptor,
+        # and whether there is inter kernel dependencies. We therefore must safely allocate after
+        # Waiting on the prior kernel to finish.
+        a_desc = tl.make_tensor_descriptor(
+            a_ptr,
+            shape=[M, K],
+            strides=[K, 1],
+            block_shape=[BLOCK_SIZE_M, BLOCK_SIZE_K],
+        )
+        b_desc = tl.make_tensor_descriptor(
+            b_ptr,
+            shape=[N, K],
+            strides=[K, 1],
+            block_shape=[BLOCK_SIZE_N, BLOCK_SIZE_K],
+        )
+        c_desc = tl.make_tensor_descriptor(
+            c_ptr,
+            shape=[M, N],
+            strides=[N, 1],
+            block_shape=[BLOCK_SIZE_M, BLOCK_SIZE_N],
+        )
+
+    for ki in range(k_tiles):
+        a = None
+        b = None
+        if USE_TMA:
+            offs_k = ki * BLOCK_SIZE_K
+            a = a_desc.load([offs_am, offs_k])
+            b = b_desc.load([offs_bn, offs_k])
+        else:
+            a = tl.load(a_ptrs, mask=offs_k[None, :] < K - ki * BLOCK_SIZE_K, other=0.0)
+            b = tl.load(b_ptrs, mask=offs_k[None, :] < K - ki * BLOCK_SIZE_K, other=0.0)
+            a_ptrs += BLOCK_SIZE_K
+            b_ptrs += BLOCK_SIZE_K
+        accumulator = tl.dot(a, b.T, accumulator)
+
+    if USE_GDC:
+        # GDC launch dependents is used to launch dependent kernels.
+        # Once GDC launch it is possible for the next kernel to begin if
+        # there are enough resources.
+        tl.extra.cuda.gdc_launch_dependents()
+
+    c = accumulator.to(dtype)
+    if USE_TMA:
+        c_desc.store([offs_am, offs_bn], c)
+    else:
+        offs_cm = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+        offs_cn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+        c_ptrs = c_ptr + N * offs_cm[:, None] + offs_cn[None, :]
+        c_mask = (offs_cm[:, None] < M) & (offs_cn[None, :] < N)
+        tl.store(c_ptrs, c, mask=c_mask)
+
+
+def matmul(a, b, launch_pdl=True):
+    # Check constraints.
+    assert a.shape[1] == b.shape[1], "Incompatible dimensions"  # b is transposed
+    assert a.dtype == b.dtype, "Incompatible dtypes"
+
+    M, K = a.shape
+    N, K = b.shape
+    dtype = a.dtype
+
+    c = torch.empty((M, N), device=a.device, dtype=dtype)
+
+    # TMA descriptors require a global memory allocation
+    def alloc_fn(size: int, alignment: int, stream: Optional[int]):
+        return torch.empty(size, device="cuda", dtype=torch.int8)
+
+    triton.set_allocator(alloc_fn)
+
+    grid = lambda META: (triton.cdiv(M, META["BLOCK_SIZE_M"]) * triton.cdiv(N, META["BLOCK_SIZE_N"]), )
+    matmul_kernel[grid](
path: python/tutorials/11-programmatic-dependent-launch.py
user: login: peterbell10
body: Oh I see, the speedup comes from launching such a small size that it doesn't fill the GPU SMs. This is less interesting IMO. Would you not see an even bigger speedup from using multiple streams instead?
created_at: 2025-04-09T21:23:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,240 @@
+"""
+Programmatic Dependent Launch
+=====================
+This script demonstrates the use of programmatic dependent launch ontop of a non-persistent implementation of matrix multiplication using Triton.
+The kernels support both FP16 and FP8 data types but the FP8 implementation is only available on CUDA devices with compute capability >= 9.0.
+
+.. code-block:: bash
+    python 11-programmatic-dependent-launch.py
+"""
+
+import torch
+import triton
+import triton.language as tl
+
+from typing import Optional
+
+
+def is_cuda():
+    return triton.runtime.driver.active.get_current_target().backend == "cuda"
+
+
+def supports_tma():
+    return is_cuda() and torch.cuda.get_device_capability()[0] >= 9
+
+
+def _matmul_launch_metadata(grid, kernel, args):
+    ret = {}
+    M, N, K = args["M"], args["N"], args["K"]
+    ret["name"] = f"{kernel.name} [M={M}, N={N}, K={K}]"
+    if "c_ptr" in args:
+        bytes_per_elem = args["c_ptr"].element_size()
+    elif "c_desc_ptr" in args:
+        bytes_per_elem = 2
+    else:
+        bytes_per_elem = 1 if args["FP8_OUTPUT"] else 2
+    ret[f"flops{bytes_per_elem * 8}"] = 2. * M * N * K
+    ret["bytes"] = bytes_per_elem * (M * K + N * K + M * N)
+    return ret
+
+
+def matmul_get_configs():
+    return [
+        triton.Config({'BLOCK_SIZE_M': BM, 'BLOCK_SIZE_N': BN, "BLOCK_SIZE_K" : BK, "USE_TMA" : USE_TMA}, num_stages=s, num_warps=w) \
+        for BM in [128] \
+        for BN in [128, 256] \
+        for BK in [64, 128] \
+        for USE_TMA in [True, False] \
+        for s in ([3, 4]) \
+        for w in [4, 8] \
+    ]
+
+
+@triton.autotune(
+    configs=matmul_get_configs(),
+    key=["M", "N", "K"],
+)
+@triton.jit(launch_metadata=_matmul_launch_metadata)
+def matmul_kernel(a_ptr, b_ptr, c_ptr,  #
+                  M, N, K,  #
+                  BLOCK_SIZE_M: tl.constexpr,  #
+                  BLOCK_SIZE_N: tl.constexpr,  #
+                  BLOCK_SIZE_K: tl.constexpr,  #
+                  USE_TMA: tl.constexpr,  #
+                  USE_GDC: tl.constexpr):  #
+    # This kernel can use TMA or traditional tl.load/tl.store
+    a_desc = None
+    b_desc = None
+    c_desc = None
+    a_ptrs = None
+    b_ptrs = None
+    c_ptrs = None
+
+    dtype = c_ptr.dtype.element_ty
+    start_pid = tl.program_id(axis=0)
+    num_pid_m = tl.cdiv(M, BLOCK_SIZE_M)
+    k_tiles = tl.cdiv(K, BLOCK_SIZE_K)
+
+    pid_m = start_pid % num_pid_m
+    pid_n = start_pid // num_pid_m
+
+    if USE_TMA:
+        offs_am = pid_m * BLOCK_SIZE_M
+        offs_bn = pid_n * BLOCK_SIZE_N
+    else:
+        offs_am = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+        offs_bn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+        offs_am = tl.where(offs_am < M, offs_am, 0)
+        offs_bn = tl.where(offs_bn < N, offs_bn, 0)
+        offs_am = tl.max_contiguous(tl.multiple_of(offs_am, BLOCK_SIZE_M), BLOCK_SIZE_M)
+        offs_bn = tl.max_contiguous(tl.multiple_of(offs_bn, BLOCK_SIZE_N), BLOCK_SIZE_N)
+        offs_k = tl.arange(0, BLOCK_SIZE_K)
+        a_ptrs = a_ptr + (offs_am[:, None] * K + offs_k[None, :])
+        # B is N x K
+        b_ptrs = b_ptr + (offs_k[None, :] + offs_bn[:, None] * K)
path: python/tutorials/11-programmatic-dependent-launch.py
user: login: ThomasRaoux
body: can we simplify this and only support TMA or not TMA. Tutorials are meant to be simple and showcase a single thing. I would try to make it minimal. If you just want to make an example making a minimal test is better.
created_at: 2025-04-10T01:30:49Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,13 @@
+from triton.language import core
+
+
+@core.extern
+def gdc_wait(_builder=None):
+    return core.inline_asm_elementwise("griddepcontrol.wait; // dummy $0", "=r", [], dtype=core.int32, is_pure=False,
path: third_party/nvidia/language/cuda/gdc.py
user: login: ThomasRaoux
body: why do you need to return the dummy value?
created_at: 2025-04-10T01:32:55Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,265 @@
+"""
+Programmatic Dependent Launch
+=====================
+This script demonstrates the use of programmatic dependent launch ontop of a non-persistent implementation of matrix multiplication using Triton.
+The kernels support both FP16 and FP8 data types but the FP8 implementation is only available on CUDA devices with compute capability >= 9.0.
+
+.. code-block:: bash
+    python 11-programmatic-dependent-launch.py
+"""
+
+import torch
+import triton
+import triton.language as tl
+
+from typing import Optional
+
+
+def is_cuda():
+    return triton.runtime.driver.active.get_current_target().backend == "cuda"
+
+
+def supports_tma():
+    return is_cuda() and torch.cuda.get_device_capability()[0] >= 9
+
+
+def _matmul_launch_metadata(grid, kernel, args):
+    ret = {}
+    M, N, K = args["M"], args["N"], args["K"]
+    ret["name"] = f"{kernel.name} [M={M}, N={N}, K={K}]"
+    if "c_ptr" in args:
+        bytes_per_elem = args["c_ptr"].element_size()
+    elif "c_desc_ptr" in args:
+        bytes_per_elem = 2
+    else:
+        bytes_per_elem = 1 if args["FP8_OUTPUT"] else 2
+    ret[f"flops{bytes_per_elem * 8}"] = 2. * M * N * K
+    ret["bytes"] = bytes_per_elem * (M * K + N * K + M * N)
+    return ret
+
+
+def matmul_get_configs():
+    return [
+        triton.Config({'BLOCK_SIZE_M': BM, 'BLOCK_SIZE_N': BN, "BLOCK_SIZE_K" : BK, "USE_TMA" : USE_TMA}, num_stages=s, num_warps=w) \
+        for BM in [128] \
+        for BN in [128, 256] \
+        for BK in [64, 128] \
+        for USE_TMA in [True, False] \
+        for s in ([3, 4]) \
+        for w in [4, 8] \
+    ]
+
+
+@triton.autotune(
+    configs=matmul_get_configs(),
+    key=["M", "N", "K"],
+)
+@triton.jit(launch_metadata=_matmul_launch_metadata)
+def matmul_kernel(a_ptr, b_ptr, c_ptr,  #
+                  M, N, K,  #
+                  BLOCK_SIZE_M: tl.constexpr,  #
+                  BLOCK_SIZE_N: tl.constexpr,  #
+                  BLOCK_SIZE_K: tl.constexpr,  #
+                  USE_TMA: tl.constexpr,  #
+                  USE_GDC: tl.constexpr):  #
+    # This kernel can use TMA or traditional tl.load/tl.store
+    a_desc = None
+    b_desc = None
+    c_desc = None
+    a_ptrs = None
+    b_ptrs = None
+    c_ptrs = None
+
+    dtype = c_ptr.dtype.element_ty
+    start_pid = tl.program_id(axis=0)
+    num_pid_m = tl.cdiv(M, BLOCK_SIZE_M)
+    k_tiles = tl.cdiv(K, BLOCK_SIZE_K)
+
+    pid_m = start_pid % num_pid_m
+    pid_n = start_pid // num_pid_m
+
+    if USE_TMA:
+        offs_am = pid_m * BLOCK_SIZE_M
+        offs_bn = pid_n * BLOCK_SIZE_N
+    else:
+        offs_am = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+        offs_bn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+        offs_am = tl.where(offs_am < M, offs_am, 0)
+        offs_bn = tl.where(offs_bn < N, offs_bn, 0)
+        offs_am = tl.max_contiguous(tl.multiple_of(offs_am, BLOCK_SIZE_M), BLOCK_SIZE_M)
+        offs_bn = tl.max_contiguous(tl.multiple_of(offs_bn, BLOCK_SIZE_N), BLOCK_SIZE_N)
+        offs_k = tl.arange(0, BLOCK_SIZE_K)
+        a_ptrs = a_ptr + (offs_am[:, None] * K + offs_k[None, :])
+        # B is N x K
+        b_ptrs = b_ptr + (offs_k[None, :] + offs_bn[:, None] * K)
+
+    accumulator = tl.zeros((BLOCK_SIZE_M, BLOCK_SIZE_N), dtype=tl.float32)
+
+    if USE_GDC:
+        # GDC wait is used to wait for the prior kernel to complete before continuing.
+        # If we utilize Programmatic Dependent Launch, we must wait on the prior kernel
+        # to complete in case a or b are written to by the prior kernel.
+        # This is done to prevent races.
+        tl.extra.cuda.gdc_wait()
+
+    if USE_TMA:
+        # We have no guarentees about what memory may be passed for our make tensor descriptor,
+        # and whether there is inter kernel dependencies. We therefore must safely allocate after
+        # Waiting on the prior kernel to finish.
+        a_desc = tl.make_tensor_descriptor(
+            a_ptr,
+            shape=[M, K],
+            strides=[K, 1],
+            block_shape=[BLOCK_SIZE_M, BLOCK_SIZE_K],
+        )
+        b_desc = tl.make_tensor_descriptor(
+            b_ptr,
+            shape=[N, K],
+            strides=[K, 1],
+            block_shape=[BLOCK_SIZE_N, BLOCK_SIZE_K],
+        )
+        c_desc = tl.make_tensor_descriptor(
+            c_ptr,
+            shape=[M, N],
+            strides=[N, 1],
+            block_shape=[BLOCK_SIZE_M, BLOCK_SIZE_N],
+        )
+
+    for ki in range(k_tiles):
+        a = None
+        b = None
+        if USE_TMA:
+            offs_k = ki * BLOCK_SIZE_K
+            a = a_desc.load([offs_am, offs_k])
+            b = b_desc.load([offs_bn, offs_k])
+        else:
+            a = tl.load(a_ptrs, mask=offs_k[None, :] < K - ki * BLOCK_SIZE_K, other=0.0)
+            b = tl.load(b_ptrs, mask=offs_k[None, :] < K - ki * BLOCK_SIZE_K, other=0.0)
+            a_ptrs += BLOCK_SIZE_K
+            b_ptrs += BLOCK_SIZE_K
+        accumulator = tl.dot(a, b.T, accumulator)
+
+    if USE_GDC:
+        # GDC launch dependents is used to launch dependent kernels.
+        # Once GDC launch it is possible for the next kernel to begin if
+        # there are enough resources.
+        tl.extra.cuda.gdc_launch_dependents()
+
+    c = accumulator.to(dtype)
+    if USE_TMA:
+        c_desc.store([offs_am, offs_bn], c)
+    else:
+        offs_cm = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+        offs_cn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+        c_ptrs = c_ptr + N * offs_cm[:, None] + offs_cn[None, :]
+        c_mask = (offs_cm[:, None] < M) & (offs_cn[None, :] < N)
+        tl.store(c_ptrs, c, mask=c_mask)
+
+
+def matmul(a, b, launch_pdl=True):
+    # Check constraints.
+    assert a.shape[1] == b.shape[1], "Incompatible dimensions"  # b is transposed
+    assert a.dtype == b.dtype, "Incompatible dtypes"
+
+    M, K = a.shape
+    N, K = b.shape
+    dtype = a.dtype
+
+    c = torch.empty((M, N), device=a.device, dtype=dtype)
+
+    # TMA descriptors require a global memory allocation
+    def alloc_fn(size: int, alignment: int, stream: Optional[int]):
+        return torch.empty(size, device="cuda", dtype=torch.int8)
+
+    triton.set_allocator(alloc_fn)
+
+    grid = lambda META: (triton.cdiv(M, META["BLOCK_SIZE_M"]) * triton.cdiv(N, META["BLOCK_SIZE_N"]), )
+    matmul_kernel[grid](
path: python/tutorials/11-programmatic-dependent-launch.py
user: login: depaulmillz
body: The issue with multiple streams is if there are dependencies between kernels. For example if GEMM 1 feeds into GEMM 2, both GEMMs cannot run in parallel. PDL allows for these to overlap for parts of the kernel that are independent (e.g., prologue of kernel 2 does not depend on kernel one). For small back to back GEMMs (memory bandwidth bound) that are only a few waves, this is a very useful optimization to have.
created_at: 2025-04-11T00:40:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,265 @@
+"""
+Programmatic Dependent Launch
+=====================
+This script demonstrates the use of programmatic dependent launch ontop of a non-persistent implementation of matrix multiplication using Triton.
+The kernels support both FP16 and FP8 data types but the FP8 implementation is only available on CUDA devices with compute capability >= 9.0.
+
+.. code-block:: bash
+    python 11-programmatic-dependent-launch.py
+"""
+
+import torch
+import triton
+import triton.language as tl
+
+from typing import Optional
+
+
+def is_cuda():
+    return triton.runtime.driver.active.get_current_target().backend == "cuda"
+
+
+def supports_tma():
+    return is_cuda() and torch.cuda.get_device_capability()[0] >= 9
+
+
+def _matmul_launch_metadata(grid, kernel, args):
+    ret = {}
+    M, N, K = args["M"], args["N"], args["K"]
+    ret["name"] = f"{kernel.name} [M={M}, N={N}, K={K}]"
+    if "c_ptr" in args:
+        bytes_per_elem = args["c_ptr"].element_size()
+    elif "c_desc_ptr" in args:
+        bytes_per_elem = 2
+    else:
+        bytes_per_elem = 1 if args["FP8_OUTPUT"] else 2
+    ret[f"flops{bytes_per_elem * 8}"] = 2. * M * N * K
+    ret["bytes"] = bytes_per_elem * (M * K + N * K + M * N)
+    return ret
+
+
+def matmul_get_configs():
+    return [
+        triton.Config({'BLOCK_SIZE_M': BM, 'BLOCK_SIZE_N': BN, "BLOCK_SIZE_K" : BK, "USE_TMA" : USE_TMA}, num_stages=s, num_warps=w) \
+        for BM in [128] \
+        for BN in [128, 256] \
+        for BK in [64, 128] \
+        for USE_TMA in [True, False] \
+        for s in ([3, 4]) \
+        for w in [4, 8] \
+    ]
+
+
+@triton.autotune(
+    configs=matmul_get_configs(),
+    key=["M", "N", "K"],
+)
+@triton.jit(launch_metadata=_matmul_launch_metadata)
+def matmul_kernel(a_ptr, b_ptr, c_ptr,  #
+                  M, N, K,  #
+                  BLOCK_SIZE_M: tl.constexpr,  #
+                  BLOCK_SIZE_N: tl.constexpr,  #
+                  BLOCK_SIZE_K: tl.constexpr,  #
+                  USE_TMA: tl.constexpr,  #
+                  USE_GDC: tl.constexpr):  #
+    # This kernel can use TMA or traditional tl.load/tl.store
+    a_desc = None
+    b_desc = None
+    c_desc = None
+    a_ptrs = None
+    b_ptrs = None
+    c_ptrs = None
+
+    dtype = c_ptr.dtype.element_ty
+    start_pid = tl.program_id(axis=0)
+    num_pid_m = tl.cdiv(M, BLOCK_SIZE_M)
+    k_tiles = tl.cdiv(K, BLOCK_SIZE_K)
+
+    pid_m = start_pid % num_pid_m
+    pid_n = start_pid // num_pid_m
+
+    if USE_TMA:
+        offs_am = pid_m * BLOCK_SIZE_M
+        offs_bn = pid_n * BLOCK_SIZE_N
+    else:
+        offs_am = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+        offs_bn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+        offs_am = tl.where(offs_am < M, offs_am, 0)
+        offs_bn = tl.where(offs_bn < N, offs_bn, 0)
+        offs_am = tl.max_contiguous(tl.multiple_of(offs_am, BLOCK_SIZE_M), BLOCK_SIZE_M)
+        offs_bn = tl.max_contiguous(tl.multiple_of(offs_bn, BLOCK_SIZE_N), BLOCK_SIZE_N)
+        offs_k = tl.arange(0, BLOCK_SIZE_K)
+        a_ptrs = a_ptr + (offs_am[:, None] * K + offs_k[None, :])
+        # B is N x K
+        b_ptrs = b_ptr + (offs_k[None, :] + offs_bn[:, None] * K)
+
+    accumulator = tl.zeros((BLOCK_SIZE_M, BLOCK_SIZE_N), dtype=tl.float32)
+
+    if USE_GDC:
+        # GDC wait is used to wait for the prior kernel to complete before continuing.
+        # If we utilize Programmatic Dependent Launch, we must wait on the prior kernel
+        # to complete in case a or b are written to by the prior kernel.
+        # This is done to prevent races.
+        tl.extra.cuda.gdc_wait()
+
+    if USE_TMA:
+        # We have no guarentees about what memory may be passed for our make tensor descriptor,
+        # and whether there is inter kernel dependencies. We therefore must safely allocate after
+        # Waiting on the prior kernel to finish.
+        a_desc = tl.make_tensor_descriptor(
+            a_ptr,
+            shape=[M, K],
+            strides=[K, 1],
+            block_shape=[BLOCK_SIZE_M, BLOCK_SIZE_K],
+        )
+        b_desc = tl.make_tensor_descriptor(
+            b_ptr,
+            shape=[N, K],
+            strides=[K, 1],
+            block_shape=[BLOCK_SIZE_N, BLOCK_SIZE_K],
+        )
+        c_desc = tl.make_tensor_descriptor(
+            c_ptr,
+            shape=[M, N],
+            strides=[N, 1],
+            block_shape=[BLOCK_SIZE_M, BLOCK_SIZE_N],
+        )
+
+    for ki in range(k_tiles):
+        a = None
+        b = None
+        if USE_TMA:
+            offs_k = ki * BLOCK_SIZE_K
+            a = a_desc.load([offs_am, offs_k])
+            b = b_desc.load([offs_bn, offs_k])
+        else:
+            a = tl.load(a_ptrs, mask=offs_k[None, :] < K - ki * BLOCK_SIZE_K, other=0.0)
+            b = tl.load(b_ptrs, mask=offs_k[None, :] < K - ki * BLOCK_SIZE_K, other=0.0)
+            a_ptrs += BLOCK_SIZE_K
+            b_ptrs += BLOCK_SIZE_K
+        accumulator = tl.dot(a, b.T, accumulator)
+
+    if USE_GDC:
+        # GDC launch dependents is used to launch dependent kernels.
+        # Once GDC launch it is possible for the next kernel to begin if
+        # there are enough resources.
+        tl.extra.cuda.gdc_launch_dependents()
+
+    c = accumulator.to(dtype)
+    if USE_TMA:
+        c_desc.store([offs_am, offs_bn], c)
+    else:
+        offs_cm = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+        offs_cn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+        c_ptrs = c_ptr + N * offs_cm[:, None] + offs_cn[None, :]
+        c_mask = (offs_cm[:, None] < M) & (offs_cn[None, :] < N)
+        tl.store(c_ptrs, c, mask=c_mask)
+
+
+def matmul(a, b, launch_pdl=True):
+    # Check constraints.
+    assert a.shape[1] == b.shape[1], "Incompatible dimensions"  # b is transposed
+    assert a.dtype == b.dtype, "Incompatible dtypes"
+
+    M, K = a.shape
+    N, K = b.shape
+    dtype = a.dtype
+
+    c = torch.empty((M, N), device=a.device, dtype=dtype)
+
+    # TMA descriptors require a global memory allocation
+    def alloc_fn(size: int, alignment: int, stream: Optional[int]):
+        return torch.empty(size, device="cuda", dtype=torch.int8)
+
+    triton.set_allocator(alloc_fn)
+
+    grid = lambda META: (triton.cdiv(M, META["BLOCK_SIZE_M"]) * triton.cdiv(N, META["BLOCK_SIZE_N"]), )
+    matmul_kernel[grid](
path: python/tutorials/11-programmatic-dependent-launch.py
user: login: depaulmillz
body: This is also useful to deal with wave quantization.
created_at: 2025-04-11T00:41:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,13 @@
+from triton.language import core
+
+
+@core.extern
+def gdc_wait(_builder=None):
+    return core.inline_asm_elementwise("griddepcontrol.wait; // dummy $0", "=r", [], dtype=core.int32, is_pure=False,
path: third_party/nvidia/language/cuda/gdc.py
user: login: depaulmillz
body: The `inline_asm_elementwise` requires a return value, so I return a dummy value.
created_at: 2025-04-11T17:01:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,13 @@
+from triton.language import core
+
+
+@core.extern
+def gdc_wait(_builder=None):
+    return core.inline_asm_elementwise("griddepcontrol.wait; // dummy $0", "=r", [], dtype=core.int32, is_pure=False,
path: third_party/nvidia/language/cuda/gdc.py
user: login: depaulmillz
body: I'm not sure if the limitation is in the core op or there is an assumption in the compiler for the inline_asm_elementwise ops.
created_at: 2025-04-11T17:03:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,13 @@
+from triton.language import core
+
+
+@core.extern
+def gdc_wait(_builder=None):
+    return core.inline_asm_elementwise("griddepcontrol.wait; // dummy $0", "=r", [], dtype=core.int32, is_pure=False,
path: third_party/nvidia/language/cuda/gdc.py
user: login: ThomasRaoux
body: I meant you don't need to return from gdc_wait, could be:
```
@core.extern
def gdc_wait(_builder=None):
   core.inline_asm_elementwise("griddepcontrol.wait; // dummy $0", "=r", [], dtype=core.int32, is_pure=False,
```
created_at: 2025-04-11T17:15:11Z
author_association: COLLABORATOR
diff_hunk: @@ -0,0 +1,13 @@
+from triton.language import core
+
+
+@core.extern
+def gdc_wait(_builder=None):
+    return core.inline_asm_elementwise("griddepcontrol.wait; // dummy $0", "=r", [], dtype=core.int32, is_pure=False,
path: third_party/nvidia/language/cuda/gdc.py
user: login: depaulmillz
body: I just changed this to not return thanks for catching it.
created_at: 2025-04-16T16:38:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,240 @@
+"""
+Programmatic Dependent Launch
+=====================
+This script demonstrates the use of programmatic dependent launch ontop of a non-persistent implementation of matrix multiplication using Triton.
+The kernels support both FP16 and FP8 data types but the FP8 implementation is only available on CUDA devices with compute capability >= 9.0.
+
+.. code-block:: bash
+    python 11-programmatic-dependent-launch.py
+"""
+
+import torch
+import triton
+import triton.language as tl
+
+from typing import Optional
+
+
+def is_cuda():
+    return triton.runtime.driver.active.get_current_target().backend == "cuda"
+
+
+def supports_tma():
+    return is_cuda() and torch.cuda.get_device_capability()[0] >= 9
+
+
+def _matmul_launch_metadata(grid, kernel, args):
+    ret = {}
+    M, N, K = args["M"], args["N"], args["K"]
+    ret["name"] = f"{kernel.name} [M={M}, N={N}, K={K}]"
+    if "c_ptr" in args:
+        bytes_per_elem = args["c_ptr"].element_size()
+    elif "c_desc_ptr" in args:
+        bytes_per_elem = 2
+    else:
+        bytes_per_elem = 1 if args["FP8_OUTPUT"] else 2
+    ret[f"flops{bytes_per_elem * 8}"] = 2. * M * N * K
+    ret["bytes"] = bytes_per_elem * (M * K + N * K + M * N)
+    return ret
+
+
+def matmul_get_configs():
+    return [
+        triton.Config({'BLOCK_SIZE_M': BM, 'BLOCK_SIZE_N': BN, "BLOCK_SIZE_K" : BK, "USE_TMA" : USE_TMA}, num_stages=s, num_warps=w) \
+        for BM in [128] \
+        for BN in [128, 256] \
+        for BK in [64, 128] \
+        for USE_TMA in [True, False] \
+        for s in ([3, 4]) \
+        for w in [4, 8] \
+    ]
+
+
+@triton.autotune(
+    configs=matmul_get_configs(),
+    key=["M", "N", "K"],
+)
+@triton.jit(launch_metadata=_matmul_launch_metadata)
+def matmul_kernel(a_ptr, b_ptr, c_ptr,  #
+                  M, N, K,  #
+                  BLOCK_SIZE_M: tl.constexpr,  #
+                  BLOCK_SIZE_N: tl.constexpr,  #
+                  BLOCK_SIZE_K: tl.constexpr,  #
+                  USE_TMA: tl.constexpr,  #
+                  USE_GDC: tl.constexpr):  #
+    # This kernel can use TMA or traditional tl.load/tl.store
+    a_desc = None
+    b_desc = None
+    c_desc = None
+    a_ptrs = None
+    b_ptrs = None
+    c_ptrs = None
+
+    dtype = c_ptr.dtype.element_ty
+    start_pid = tl.program_id(axis=0)
+    num_pid_m = tl.cdiv(M, BLOCK_SIZE_M)
+    k_tiles = tl.cdiv(K, BLOCK_SIZE_K)
+
+    pid_m = start_pid % num_pid_m
+    pid_n = start_pid // num_pid_m
+
+    if USE_TMA:
+        offs_am = pid_m * BLOCK_SIZE_M
+        offs_bn = pid_n * BLOCK_SIZE_N
+    else:
+        offs_am = pid_m * BLOCK_SIZE_M + tl.arange(0, BLOCK_SIZE_M)
+        offs_bn = pid_n * BLOCK_SIZE_N + tl.arange(0, BLOCK_SIZE_N)
+        offs_am = tl.where(offs_am < M, offs_am, 0)
+        offs_bn = tl.where(offs_bn < N, offs_bn, 0)
+        offs_am = tl.max_contiguous(tl.multiple_of(offs_am, BLOCK_SIZE_M), BLOCK_SIZE_M)
+        offs_bn = tl.max_contiguous(tl.multiple_of(offs_bn, BLOCK_SIZE_N), BLOCK_SIZE_N)
+        offs_k = tl.arange(0, BLOCK_SIZE_K)
+        a_ptrs = a_ptr + (offs_am[:, None] * K + offs_k[None, :])
+        # B is N x K
+        b_ptrs = b_ptr + (offs_k[None, :] + offs_bn[:, None] * K)
path: python/tutorials/11-programmatic-dependent-launch.py
user: login: depaulmillz
body: I simplified the tutorial to just reuse the vector add tutorial with and without PDL.
created_at: 2025-04-16T16:39:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,128 @@
+"""
+Programmatic Dependent Launch
+=====================
+This script demonstrates the use of programmatic dependent launch ontop of the vector-add example using Triton.
+
+.. code-block:: bash
+    python 11-programmatic-dependent-launch.py
+"""
+
+import torch
+import triton
+import triton.language as tl
+
+
+def is_cuda():
+    return triton.runtime.driver.active.get_current_target().backend == "cuda"
+
+
+def supports_pdl():
+    return is_cuda() and torch.cuda.get_device_capability()[0] >= 9
+
+
+# In this example
+@triton.jit
+def add_kernel(x_ptr,  #
+               y_ptr,  #
+               output_ptr,  #
+               n_elements,  #
+               BLOCK_SIZE: tl.constexpr,  #
+               USE_GDC: tl.constexpr,  #
+               ):
+    pid = tl.program_id(axis=0)
+    block_start = pid * BLOCK_SIZE
+    offsets = block_start + tl.arange(0, BLOCK_SIZE)
+    mask = offsets < n_elements
+    if USE_GDC:
+        # GDC wait is used to wait for the prior kernel to complete before continuing.
+        # If we utilize Programmatic Dependent Launch, we must wait on the prior kernel
+        # to complete in case a or b are written to by the prior kernel.
+        # This is done to prevent races.
+        tl.extra.cuda.gdc_wait()
+
+    x = tl.load(x_ptr + offsets, mask=mask)
+    y = tl.load(y_ptr + offsets, mask=mask)
+    if USE_GDC:
+        # GDC launch dependents is used to launch dependent kernels.
+        # Once GDC launch it is possible for the next kernel to begin if
+        # there are enough resources.
+        tl.extra.cuda.gdc_launch_dependents()
+    output = x + y
+    tl.store(output_ptr + offsets, output, mask=mask)
+
+
+def add(x: torch.Tensor, y: torch.Tensor, launch_pdl: bool = True):
+    output = torch.empty_like(x)
+    assert x.device == y.device and output.device == x.device
+    n_elements = output.numel()
+    grid = lambda meta: (triton.cdiv(n_elements, meta['BLOCK_SIZE']), )
+    add_kernel[grid](
+        x, y, output, n_elements, BLOCK_SIZE=1024,
+        USE_GDC=launch_pdl,  # set constexpr in kernel to use grid dependence control
+        launch_pdl=launch_pdl,  # launch kernel with PDL flag set enabled
+    )
+    return output
+
+
+def validate(n_elements):
+    x = torch.rand(n_elements, device="cuda", dtype=torch.float32)
+    y = torch.rand(n_elements, device="cuda", dtype=torch.float32)
+
+    torch_result = x + y
+    add_result = add(x, y)
+
+    torch_vs_add = "✅" if torch.allclose(torch_result, add_result, atol=1.0) else "❌"
+    print(f"Number of Elements={n_elements} verification naive vs: ", end="")
+    print(f"add: {torch_vs_add}")
+
+
+@triton.testing.perf_report(
+    triton.testing.Benchmark(
+        x_names=["size"],
+        x_vals=[2**i for i in range(12, 28, 1)],
+        x_log=False,
+        line_arg="provider",
+        line_vals=["pdl-fp32", "fp32"],
+        line_names=["PDL", "No PDL"],
+        styles=[("red", "-"), ("blue", "-")],
+        ylabel='GB/s',
+        plot_name="pdl-performance",
+        args={},
+    ))
+def benchmark(size, provider):
+
+    x = list()
+    y = list()
+
+    for _ in range(100):
+        x.append(torch.rand(size, device="cuda", dtype=torch.float32))
+        y.append(torch.rand(size, device="cuda", dtype=torch.float32))
path: python/tutorials/11-programmatic-dependent-launch.py
user: login: peterbell10
body: This OOMs on an H100 with 80GB of gmem, can we just have one `x` and `y`?
created_at: 2025-04-16T17:13:38Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,128 @@
+"""
+Programmatic Dependent Launch
+=====================
+This script demonstrates the use of programmatic dependent launch ontop of the vector-add example using Triton.
+
+.. code-block:: bash
+    python 11-programmatic-dependent-launch.py
+"""
+
+import torch
+import triton
+import triton.language as tl
+
+
+def is_cuda():
+    return triton.runtime.driver.active.get_current_target().backend == "cuda"
+
+
+def supports_pdl():
+    return is_cuda() and torch.cuda.get_device_capability()[0] >= 9
+
+
+# In this example
+@triton.jit
+def add_kernel(x_ptr,  #
+               y_ptr,  #
+               output_ptr,  #
+               n_elements,  #
+               BLOCK_SIZE: tl.constexpr,  #
+               USE_GDC: tl.constexpr,  #
+               ):
+    pid = tl.program_id(axis=0)
+    block_start = pid * BLOCK_SIZE
+    offsets = block_start + tl.arange(0, BLOCK_SIZE)
+    mask = offsets < n_elements
+    if USE_GDC:
+        # GDC wait is used to wait for the prior kernel to complete before continuing.
+        # If we utilize Programmatic Dependent Launch, we must wait on the prior kernel
+        # to complete in case a or b are written to by the prior kernel.
+        # This is done to prevent races.
+        tl.extra.cuda.gdc_wait()
+
+    x = tl.load(x_ptr + offsets, mask=mask)
+    y = tl.load(y_ptr + offsets, mask=mask)
+    if USE_GDC:
+        # GDC launch dependents is used to launch dependent kernels.
+        # Once GDC launch it is possible for the next kernel to begin if
+        # there are enough resources.
+        tl.extra.cuda.gdc_launch_dependents()
+    output = x + y
+    tl.store(output_ptr + offsets, output, mask=mask)
+
+
+def add(x: torch.Tensor, y: torch.Tensor, launch_pdl: bool = True):
+    output = torch.empty_like(x)
+    assert x.device == y.device and output.device == x.device
+    n_elements = output.numel()
+    grid = lambda meta: (triton.cdiv(n_elements, meta['BLOCK_SIZE']), )
+    add_kernel[grid](
+        x, y, output, n_elements, BLOCK_SIZE=1024,
+        USE_GDC=launch_pdl,  # set constexpr in kernel to use grid dependence control
+        launch_pdl=launch_pdl,  # launch kernel with PDL flag set enabled
+    )
+    return output
+
+
+def validate(n_elements):
+    x = torch.rand(n_elements, device="cuda", dtype=torch.float32)
+    y = torch.rand(n_elements, device="cuda", dtype=torch.float32)
+
+    torch_result = x + y
+    add_result = add(x, y)
+
+    torch_vs_add = "✅" if torch.allclose(torch_result, add_result, atol=1.0) else "❌"
+    print(f"Number of Elements={n_elements} verification naive vs: ", end="")
+    print(f"add: {torch_vs_add}")
+
+
+@triton.testing.perf_report(
+    triton.testing.Benchmark(
+        x_names=["size"],
+        x_vals=[2**i for i in range(12, 28, 1)],
+        x_log=False,
+        line_arg="provider",
+        line_vals=["pdl-fp32", "fp32"],
+        line_names=["PDL", "No PDL"],
+        styles=[("red", "-"), ("blue", "-")],
+        ylabel='GB/s',
+        plot_name="pdl-performance",
+        args={},
+    ))
+def benchmark(size, provider):
+
+    x = list()
+    y = list()
+
+    for _ in range(100):
+        x.append(torch.rand(size, device="cuda", dtype=torch.float32))
+        y.append(torch.rand(size, device="cuda", dtype=torch.float32))
+
+    start = torch.cuda.Event(enable_timing=True)
+    end = torch.cuda.Event(enable_timing=True)
+
+    g = torch.cuda.CUDAGraph()
+    i = 0
+    with torch.cuda.graph(g):
+        for _ in range(1000):  # 1000 runs
+            add(x[i], y[i], "pdl" in provider)
+            i = (i + 1) % 100
+
+    g.replay()
+    start.record()
+    g.replay()
+    end.record()
+    torch.cuda.synchronize()
+    ms = start.elapsed_time(end) / 1000.0
path: python/tutorials/11-programmatic-dependent-launch.py
user: login: peterbell10
body: Can you use `do_bench_cudagraph` instead? The ratio of boiler plate to relevant code is pretty bad here.
created_at: 2025-04-16T17:14:08Z
author_association: CONTRIBUTOR
diff_hunk: @@ -271,15 +271,40 @@ def format_of(ty):
   return cuLaunchKernelExHandle;
 }}
 
-static void _launch(int gridX, int gridY, int gridZ, int num_warps, int num_ctas, int launch_cooperative_grid, int clusterDimX, int clusterDimY, int clusterDimZ, int shared_memory, CUstream stream, CUfunction function, CUdeviceptr global_scratch{', ' + arg_decls if len(arg_decls) > 0 else ''}) {{
+static void _launch(int gridX, int gridY, int gridZ, int num_warps, int num_ctas, int launch_cooperative_grid, int launch_pdl, int clusterDimX, int clusterDimY, int clusterDimZ, int shared_memory, CUstream stream, CUfunction function, CUdeviceptr global_scratch{', ' + arg_decls if len(arg_decls) > 0 else ''}) {{
   void *params[] = {{ {', '.join(params)} }};
   if (gridX*gridY*gridZ > 0) {{
-    if ((num_ctas == 1) && (0 == launch_cooperative_grid)) {{
+    if ((num_ctas == 1) && (0 == launch_cooperative_grid) && (0 == launch_pdl)) {{
       CUDA_CHECK(cuLaunchKernel(function, gridX, gridY, gridZ, 32*num_warps, 1, 1, shared_memory, stream, params, 0));
-    }} else if ((num_ctas == 1) && (0 != launch_cooperative_grid)) {{
+    }} else if ((num_ctas == 1) && (0 == launch_cooperative_grid)) {{
path: third_party/nvidia/backend/driver.py
user: login: peterbell10
body: This if-else chain is getting pretty long. Can we just unconditionally pass all the parameters even if they're the default value?
created_at: 2025-04-16T17:20:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -625,7 +658,8 @@ def __call__(self, gridX, gridY, gridZ, stream, function, *args):
             global_scratch = _allocation._allocator(alloc_size, self.global_scratch_align, stream)
         else:
             global_scratch = None
-        self.launch(gridX, gridY, gridZ, stream, function, self.launch_cooperative_grid, global_scratch, *args)
+        self.launch(gridX, gridY, gridZ, stream, function, self.launch_cooperative_grid, self.launch_pdl,
path: third_party/nvidia/backend/driver.py
user: login: peterbell10
body: Please update `wrap_handle_tensordesc` to reflect the increase in meta-arguments.
created_at: 2025-04-16T17:20:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,128 @@
+"""
+Programmatic Dependent Launch
+=====================
+This script demonstrates the use of programmatic dependent launch ontop of the vector-add example using Triton.
+
+.. code-block:: bash
+    python 11-programmatic-dependent-launch.py
+"""
+
+import torch
+import triton
+import triton.language as tl
+
+
+def is_cuda():
+    return triton.runtime.driver.active.get_current_target().backend == "cuda"
+
+
+def supports_pdl():
+    return is_cuda() and torch.cuda.get_device_capability()[0] >= 9
+
+
+# In this example
+@triton.jit
+def add_kernel(x_ptr,  #
+               y_ptr,  #
+               output_ptr,  #
+               n_elements,  #
+               BLOCK_SIZE: tl.constexpr,  #
+               USE_GDC: tl.constexpr,  #
+               ):
+    pid = tl.program_id(axis=0)
+    block_start = pid * BLOCK_SIZE
+    offsets = block_start + tl.arange(0, BLOCK_SIZE)
+    mask = offsets < n_elements
+    if USE_GDC:
+        # GDC wait is used to wait for the prior kernel to complete before continuing.
+        # If we utilize Programmatic Dependent Launch, we must wait on the prior kernel
+        # to complete in case a or b are written to by the prior kernel.
+        # This is done to prevent races.
+        tl.extra.cuda.gdc_wait()
+
+    x = tl.load(x_ptr + offsets, mask=mask)
+    y = tl.load(y_ptr + offsets, mask=mask)
+    if USE_GDC:
+        # GDC launch dependents is used to launch dependent kernels.
+        # Once GDC launch it is possible for the next kernel to begin if
+        # there are enough resources.
+        tl.extra.cuda.gdc_launch_dependents()
+    output = x + y
+    tl.store(output_ptr + offsets, output, mask=mask)
+
+
+def add(x: torch.Tensor, y: torch.Tensor, launch_pdl: bool = True):
+    output = torch.empty_like(x)
+    assert x.device == y.device and output.device == x.device
+    n_elements = output.numel()
+    grid = lambda meta: (triton.cdiv(n_elements, meta['BLOCK_SIZE']), )
+    add_kernel[grid](
+        x, y, output, n_elements, BLOCK_SIZE=1024,
+        USE_GDC=launch_pdl,  # set constexpr in kernel to use grid dependence control
+        launch_pdl=launch_pdl,  # launch kernel with PDL flag set enabled
+    )
+    return output
+
+
+def validate(n_elements):
+    x = torch.rand(n_elements, device="cuda", dtype=torch.float32)
+    y = torch.rand(n_elements, device="cuda", dtype=torch.float32)
+
+    torch_result = x + y
+    add_result = add(x, y)
+
+    torch_vs_add = "✅" if torch.allclose(torch_result, add_result, atol=1.0) else "❌"
+    print(f"Number of Elements={n_elements} verification naive vs: ", end="")
+    print(f"add: {torch_vs_add}")
+
+
+@triton.testing.perf_report(
+    triton.testing.Benchmark(
+        x_names=["size"],
+        x_vals=[2**i for i in range(12, 28, 1)],
+        x_log=False,
+        line_arg="provider",
+        line_vals=["pdl-fp32", "fp32"],
+        line_names=["PDL", "No PDL"],
+        styles=[("red", "-"), ("blue", "-")],
+        ylabel='GB/s',
+        plot_name="pdl-performance",
+        args={},
+    ))
+def benchmark(size, provider):
+
+    x = list()
+    y = list()
+
+    for _ in range(100):
+        x.append(torch.rand(size, device="cuda", dtype=torch.float32))
+        y.append(torch.rand(size, device="cuda", dtype=torch.float32))
+
+    start = torch.cuda.Event(enable_timing=True)
+    end = torch.cuda.Event(enable_timing=True)
+
+    g = torch.cuda.CUDAGraph()
+    i = 0
+    with torch.cuda.graph(g):
+        for _ in range(1000):  # 1000 runs
+            add(x[i], y[i], "pdl" in provider)
+            i = (i + 1) % 100
+
+    g.replay()
+    start.record()
+    g.replay()
+    end.record()
+    torch.cuda.synchronize()
+    ms = start.elapsed_time(end) / 1000.0
path: python/tutorials/11-programmatic-dependent-launch.py
user: login: depaulmillz
body: I just updated the code to address your concerns. `do_bench_cudagraph` doesn’t loop through multiple buffers, which could cause L2 to cache our tensors and skew the results. To address this, I've now ensured that the function uses large tensors to prevent cross-kernel L2 hits. The size is capped at 3GB, making it compatible with Geforce cards as well.

created_at: 2025-04-16T21:09:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -625,7 +658,8 @@ def __call__(self, gridX, gridY, gridZ, stream, function, *args):
             global_scratch = _allocation._allocator(alloc_size, self.global_scratch_align, stream)
         else:
             global_scratch = None
-        self.launch(gridX, gridY, gridZ, stream, function, self.launch_cooperative_grid, global_scratch, *args)
+        self.launch(gridX, gridY, gridZ, stream, function, self.launch_cooperative_grid, self.launch_pdl,
path: third_party/nvidia/backend/driver.py
user: login: depaulmillz
body: I just updated the number of meta-arguments to 12 now in the code.
created_at: 2025-04-16T21:10:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -271,15 +271,40 @@ def format_of(ty):
   return cuLaunchKernelExHandle;
 }}
 
-static void _launch(int gridX, int gridY, int gridZ, int num_warps, int num_ctas, int launch_cooperative_grid, int clusterDimX, int clusterDimY, int clusterDimZ, int shared_memory, CUstream stream, CUfunction function, CUdeviceptr global_scratch{', ' + arg_decls if len(arg_decls) > 0 else ''}) {{
+static void _launch(int gridX, int gridY, int gridZ, int num_warps, int num_ctas, int launch_cooperative_grid, int launch_pdl, int clusterDimX, int clusterDimY, int clusterDimZ, int shared_memory, CUstream stream, CUfunction function, CUdeviceptr global_scratch{', ' + arg_decls if len(arg_decls) > 0 else ''}) {{
   void *params[] = {{ {', '.join(params)} }};
   if (gridX*gridY*gridZ > 0) {{
-    if ((num_ctas == 1) && (0 == launch_cooperative_grid)) {{
+    if ((num_ctas == 1) && (0 == launch_cooperative_grid) && (0 == launch_pdl)) {{
       CUDA_CHECK(cuLaunchKernel(function, gridX, gridY, gridZ, 32*num_warps, 1, 1, shared_memory, stream, params, 0));
-    }} else if ((num_ctas == 1) && (0 != launch_cooperative_grid)) {{
+    }} else if ((num_ctas == 1) && (0 == launch_cooperative_grid)) {{
path: third_party/nvidia/backend/driver.py
user: login: depaulmillz
body: I refactored this to set up to 4 attributes and just pass the relevant ones to the kernel.
created_at: 2025-04-16T21:11:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,112 @@
+"""
+Programmatic Dependent Launch
+=====================
+This script demonstrates the use of programmatic dependent launch ontop of the vector-add example using Triton.
+
+.. code-block:: bash
+    python 11-programmatic-dependent-launch.py
+"""
+
+import torch
+import triton
+import triton.language as tl
+
+
+def is_cuda():
+    return triton.runtime.driver.active.get_current_target().backend == "cuda"
+
+
+def supports_pdl():
+    return is_cuda() and torch.cuda.get_device_capability()[0] >= 9
+
+
+# In this example
+@triton.jit
+def add_kernel(x_ptr,  #
+               y_ptr,  #
+               output_ptr,  #
+               n_elements,  #
+               BLOCK_SIZE: tl.constexpr,  #
+               USE_GDC: tl.constexpr,  #
+               ):
+    pid = tl.program_id(axis=0)
+    block_start = pid * BLOCK_SIZE
+    offsets = block_start + tl.arange(0, BLOCK_SIZE)
+    mask = offsets < n_elements
+    if USE_GDC:
+        # GDC wait is used to wait for the prior kernel to complete before continuing.
+        # If we utilize Programmatic Dependent Launch, we must wait on the prior kernel
+        # to complete in case a or b are written to by the prior kernel.
+        # This is done to prevent races.
path: python/tutorials/11-programmatic-dependent-launch.py
user: login: peterbell10
body: ```suggestion
        # GDC wait waits for ALL programs in the the prior kernel to complete before continuing.
        # This ensures any memory operations happen before the wait in program order, 
        # e.g. if the prior kernel writes to a or b the new values will be visible.
```
created_at: 2025-04-19T12:58:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,112 @@
+"""
+Programmatic Dependent Launch
+=====================
+This script demonstrates the use of programmatic dependent launch ontop of the vector-add example using Triton.
+
+.. code-block:: bash
+    python 11-programmatic-dependent-launch.py
+"""
+
+import torch
+import triton
+import triton.language as tl
+
+
+def is_cuda():
+    return triton.runtime.driver.active.get_current_target().backend == "cuda"
+
+
+def supports_pdl():
+    return is_cuda() and torch.cuda.get_device_capability()[0] >= 9
+
+
+# In this example
+@triton.jit
+def add_kernel(x_ptr,  #
+               y_ptr,  #
+               output_ptr,  #
+               n_elements,  #
+               BLOCK_SIZE: tl.constexpr,  #
+               USE_GDC: tl.constexpr,  #
+               ):
+    pid = tl.program_id(axis=0)
+    block_start = pid * BLOCK_SIZE
+    offsets = block_start + tl.arange(0, BLOCK_SIZE)
+    mask = offsets < n_elements
+    if USE_GDC:
+        # GDC wait is used to wait for the prior kernel to complete before continuing.
+        # If we utilize Programmatic Dependent Launch, we must wait on the prior kernel
+        # to complete in case a or b are written to by the prior kernel.
+        # This is done to prevent races.
+        tl.extra.cuda.gdc_wait()
+
+    x = tl.load(x_ptr + offsets, mask=mask)
+    y = tl.load(y_ptr + offsets, mask=mask)
+    if USE_GDC:
+        # GDC launch dependents is used to launch dependent kernels.
+        # Once GDC launch it is possible for the next kernel to begin if
+        # there are enough resources.
path: python/tutorials/11-programmatic-dependent-launch.py
user: login: peterbell10
body: ```suggestion
        # GDC launch dependents is used to launch dependent kernels.
        # Once GDC launch has been issued at least once on all active hardware SMs, 
        # it is possible for the next kernel to begin if there are enough resources.
        # Note this provides no additional memory-ordering guaruntees, unlike `gdc_wait`.
```
created_at: 2025-04-19T13:07:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,13 @@
+from triton.language import core
+
+
+@core.extern
+def gdc_wait(_builder=None):
+    core.inline_asm_elementwise("griddepcontrol.wait; // dummy $0", "=r", [], dtype=core.int32, is_pure=False, pack=1,
+                                _builder=_builder)
+
+
+@core.extern
+def gdc_launch_dependents(_builder=None):
path: third_party/nvidia/language/cuda/gdc.py
user: login: peterbell10
body: Can we add some reference docs for these?
created_at: 2025-04-19T13:09:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,112 @@
+"""
+Programmatic Dependent Launch
+=====================
+This script demonstrates the use of programmatic dependent launch ontop of the vector-add example using Triton.
+
+.. code-block:: bash
+    python 11-programmatic-dependent-launch.py
+"""
+
+import torch
+import triton
+import triton.language as tl
+
+
+def is_cuda():
+    return triton.runtime.driver.active.get_current_target().backend == "cuda"
+
+
+def supports_pdl():
+    return is_cuda() and torch.cuda.get_device_capability()[0] >= 9
+
+
+# In this example
+@triton.jit
+def add_kernel(x_ptr,  #
+               y_ptr,  #
+               output_ptr,  #
+               n_elements,  #
+               BLOCK_SIZE: tl.constexpr,  #
+               USE_GDC: tl.constexpr,  #
+               ):
+    pid = tl.program_id(axis=0)
+    block_start = pid * BLOCK_SIZE
+    offsets = block_start + tl.arange(0, BLOCK_SIZE)
+    mask = offsets < n_elements
+    if USE_GDC:
+        # GDC wait is used to wait for the prior kernel to complete before continuing.
+        # If we utilize Programmatic Dependent Launch, we must wait on the prior kernel
+        # to complete in case a or b are written to by the prior kernel.
+        # This is done to prevent races.
+        tl.extra.cuda.gdc_wait()
+
+    x = tl.load(x_ptr + offsets, mask=mask)
+    y = tl.load(y_ptr + offsets, mask=mask)
+    if USE_GDC:
+        # GDC launch dependents is used to launch dependent kernels.
+        # Once GDC launch it is possible for the next kernel to begin if
+        # there are enough resources.
path: python/tutorials/11-programmatic-dependent-launch.py
user: login: peterbell10
body: Is this correct? There are no suggestions of memory ordering in the docs.
https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-griddepcontrol
created_at: 2025-04-19T13:12:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,112 @@
+"""
+Programmatic Dependent Launch
+=====================
+This script demonstrates the use of programmatic dependent launch ontop of the vector-add example using Triton.
+
+.. code-block:: bash
+    python 11-programmatic-dependent-launch.py
+"""
+
+import torch
+import triton
+import triton.language as tl
+
+
+def is_cuda():
+    return triton.runtime.driver.active.get_current_target().backend == "cuda"
+
+
+def supports_pdl():
+    return is_cuda() and torch.cuda.get_device_capability()[0] >= 9
+
+
+# In this example
+@triton.jit
+def add_kernel(x_ptr,  #
+               y_ptr,  #
+               output_ptr,  #
+               n_elements,  #
+               BLOCK_SIZE: tl.constexpr,  #
+               USE_GDC: tl.constexpr,  #
+               ):
+    pid = tl.program_id(axis=0)
+    block_start = pid * BLOCK_SIZE
+    offsets = block_start + tl.arange(0, BLOCK_SIZE)
+    mask = offsets < n_elements
+    if USE_GDC:
+        # GDC wait waits for ALL programs in the the prior kernel to complete before continuing.
+        # This ensures any memory operations happen before the wait in program order, 
+        # e.g. if the prior kernel writes to a or b the new values will be visible.
+        tl.extra.cuda.gdc_wait()
+
+    x = tl.load(x_ptr + offsets, mask=mask)
+    y = tl.load(y_ptr + offsets, mask=mask)
+    if USE_GDC:
+        # GDC launch dependents is used to launch dependent kernels.
+        # Once GDC launch has been issued at least once on all active hardware SMs, 
+        # it is possible for the next kernel to begin if there are enough resources.
+        # Note this provides no additional memory-ordering guaruntees, unlike `gdc_wait`.
path: python/tutorials/11-programmatic-dependent-launch.py
user: login: peterbell10
body: Sorry, I introduced a typo
```suggestion
        # Note this provides no additional memory-ordering guarantees, unlike `gdc_wait`.
```
created_at: 2025-04-19T17:23:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,42 @@
+"""
+Grid Dependency Control (GDC) is a mechanism used when enabling programmatic dependent launch to launch and
+synchronize grids. These APIs expose GDC to the programmer.
+
+Programmatic dependent launch is supported on SM90 (Hopper) and beyond.
+For PTX reference on grid dependency control see https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-griddepcontrol.
+"""
+
+from triton.language import core
+
+
+@core.extern
+def gdc_wait(_builder=None):
+    """
+    GDC wait is a blocking instruction that waits for all instructions in a prior kernel to complete before continuing.
path: third_party/nvidia/language/cuda/gdc.py
user: login: peterbell10
body: Looks good, but you need to add this to the `rst` files for this to appear in the rendered docs.
https://github.com/triton-lang/triton/tree/main/docs/python-api
created_at: 2025-04-23T21:44:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,116 @@
+"""
+Programmatic Dependent Launch
+=====================
+This script demonstrates the use of programmatic dependent launch (PDL) ontop of the vector-add example using Triton.
+
+For CUDA reference on programmatic dependent launch see https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#programmatic-dependent-launch-and-synchronization.
+For PTX reference on programmatic dependent launch see https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-griddepcontrol.
+
+.. code-block:: bash
+    python 11-programmatic-dependent-launch.py
+"""
+
+import torch
+import triton
+import triton.language as tl
+
+
+def is_cuda():
+    return triton.runtime.driver.active.get_current_target().backend == "cuda"
+
+
+def supports_pdl():
+    return is_cuda() and torch.cuda.get_device_capability()[0] >= 9
+
+
+# In this example
+@triton.jit
+def add_kernel(x_ptr,  #
+               y_ptr,  #
+               output_ptr,  #
+               n_elements,  #
+               BLOCK_SIZE: tl.constexpr,  #
+               USE_GDC: tl.constexpr,  #
+               ):
+    pid = tl.program_id(axis=0)
+    block_start = pid * BLOCK_SIZE
+    offsets = block_start + tl.arange(0, BLOCK_SIZE)
+    mask = offsets < n_elements
+    if USE_GDC:
+        # GDC wait waits for ALL programs in the the prior kernel to complete before continuing.
+        # This ensures any memory operations happen before the wait in program order,
+        # e.g. if the prior kernel writes to x or y the new values will be visible.
+        tl.extra.cuda.gdc_wait()
+
+    x = tl.load(x_ptr + offsets, mask=mask)
+    y = tl.load(y_ptr + offsets, mask=mask)
+    if USE_GDC:
+        # GDC launch dependents hints the runtime system to launch dependent kernels.
+        # These dependent kernels must also be launched with PDL enabled.
+        # Once GDC launch has been issued by at least one thread on ALL programs or
path: python/tutorials/11-programmatic-dependent-launch.py
user: login: peterbell10
body: There are no threads in the triton programming model
created_at: 2025-04-23T21:46:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,42 @@
+"""
+Grid Dependency Control (GDC) is a mechanism used when enabling programmatic dependent launch to launch and
+synchronize grids. These APIs expose GDC to the programmer.
+
+Programmatic dependent launch is supported on SM90 (Hopper) and beyond.
+For PTX reference on grid dependency control see https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-griddepcontrol.
+"""
+
+from triton.language import core
+
+
+@core.extern
+def gdc_wait(_builder=None):
+    """
+    GDC wait is a blocking instruction that waits for all instructions in a prior kernel to complete before continuing.
+    This ensures all memory operations happening before the wait is visible to instructions after it,
+    e.g. if the prior kernel writes to address "x" the new values will be visible in this kernel after the wait.
+
+    This instruction is also safe to execute when programatic dependent launch is disabled.
+
+    See https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-griddepcontrol for more details.
+    """
+    core.inline_asm_elementwise("griddepcontrol.wait; // dummy $0", "=r", [], dtype=core.int32, is_pure=False, pack=1,
+                                _builder=_builder)
+
+
+@core.extern
+def gdc_launch_dependents(_builder=None):
+    """
+    This operation when launched with programmatic dependent launch signals that
+    the next program may launch as soon as on thread within all programs in the current kernel
path: third_party/nvidia/language/cuda/gdc.py
user: login: peterbell10
body: ```suggestion
    the next program may launch once all programs in the current kernel
```
created_at: 2025-04-23T21:48:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,42 @@
+"""
+Grid Dependency Control (GDC) is a mechanism used when enabling programmatic dependent launch to launch and
+synchronize grids. These APIs expose GDC to the programmer.
+
+Programmatic dependent launch is supported on SM90 (Hopper) and beyond.
+For PTX reference on grid dependency control see https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-griddepcontrol.
+"""
+
+from triton.language import core
+
+
+@core.extern
+def gdc_wait(_builder=None):
+    """
+    GDC wait is a blocking instruction that waits for all instructions in a prior kernel to complete before continuing.
path: third_party/nvidia/language/cuda/gdc.py
user: login: depaulmillz
body: I updated the rst.
created_at: 2025-04-29T02:51:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,116 @@
+"""
+Programmatic Dependent Launch
+=====================
+This script demonstrates the use of programmatic dependent launch (PDL) ontop of the vector-add example using Triton.
+
+For CUDA reference on programmatic dependent launch see https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#programmatic-dependent-launch-and-synchronization.
+For PTX reference on programmatic dependent launch see https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-griddepcontrol.
+
+.. code-block:: bash
+    python 11-programmatic-dependent-launch.py
+"""
+
+import torch
+import triton
+import triton.language as tl
+
+
+def is_cuda():
+    return triton.runtime.driver.active.get_current_target().backend == "cuda"
+
+
+def supports_pdl():
+    return is_cuda() and torch.cuda.get_device_capability()[0] >= 9
+
+
+# In this example
+@triton.jit
+def add_kernel(x_ptr,  #
+               y_ptr,  #
+               output_ptr,  #
+               n_elements,  #
+               BLOCK_SIZE: tl.constexpr,  #
+               USE_GDC: tl.constexpr,  #
+               ):
+    pid = tl.program_id(axis=0)
+    block_start = pid * BLOCK_SIZE
+    offsets = block_start + tl.arange(0, BLOCK_SIZE)
+    mask = offsets < n_elements
+    if USE_GDC:
+        # GDC wait waits for ALL programs in the the prior kernel to complete before continuing.
+        # This ensures any memory operations happen before the wait in program order,
+        # e.g. if the prior kernel writes to x or y the new values will be visible.
+        tl.extra.cuda.gdc_wait()
+
+    x = tl.load(x_ptr + offsets, mask=mask)
+    y = tl.load(y_ptr + offsets, mask=mask)
+    if USE_GDC:
+        # GDC launch dependents hints the runtime system to launch dependent kernels.
+        # These dependent kernels must also be launched with PDL enabled.
+        # Once GDC launch has been issued by at least one thread on ALL programs or
path: python/tutorials/11-programmatic-dependent-launch.py
user: login: depaulmillz
body: Thanks I removed the comment about threads.
created_at: 2025-04-29T02:52:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,116 @@
+"""
+Programmatic Dependent Launch
+=====================
+This script demonstrates the use of programmatic dependent launch (PDL) ontop of the vector-add example using Triton.
+
+For CUDA reference on programmatic dependent launch see https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#programmatic-dependent-launch-and-synchronization.
+For PTX reference on programmatic dependent launch see https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-griddepcontrol.
+
+.. code-block:: bash
+    python 11-programmatic-dependent-launch.py
+"""
+
+import torch
+import triton
+import triton.language as tl
+
+
+def is_cuda():
+    return triton.runtime.driver.active.get_current_target().backend == "cuda"
+
+
+def supports_pdl():
+    return is_cuda() and torch.cuda.get_device_capability()[0] >= 9
+
+
+# In this example
+@triton.jit
+def add_kernel(x_ptr,  #
+               y_ptr,  #
+               output_ptr,  #
+               n_elements,  #
+               BLOCK_SIZE: tl.constexpr,  #
+               USE_GDC: tl.constexpr,  #
+               ):
+    pid = tl.program_id(axis=0)
+    block_start = pid * BLOCK_SIZE
+    offsets = block_start + tl.arange(0, BLOCK_SIZE)
+    mask = offsets < n_elements
+    if USE_GDC:
+        # GDC wait waits for ALL programs in the the prior kernel to complete before continuing.
+        # This ensures any memory operations happen before the wait in program order,
+        # e.g. if the prior kernel writes to x or y the new values will be visible.
+        tl.extra.cuda.gdc_wait()
+
+    x = tl.load(x_ptr + offsets, mask=mask)
+    y = tl.load(y_ptr + offsets, mask=mask)
+    if USE_GDC:
+        # GDC launch dependents hints the runtime system to launch dependent kernels.
+        # These dependent kernels must also be launched with PDL enabled.
+        # Once GDC launch has been issued by at least one thread on ALL programs or
path: python/tutorials/11-programmatic-dependent-launch.py
user: login: peterbell10
body: If you have, you didn't push it yet.
created_at: 2025-04-29T11:45:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,116 @@
+"""
+Programmatic Dependent Launch
+=====================
+This script demonstrates the use of programmatic dependent launch (PDL) ontop of the vector-add example using Triton.
+
+For CUDA reference on programmatic dependent launch see https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#programmatic-dependent-launch-and-synchronization.
+For PTX reference on programmatic dependent launch see https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-griddepcontrol.
+
+.. code-block:: bash
+    python 11-programmatic-dependent-launch.py
+"""
+
+import torch
+import triton
+import triton.language as tl
+
+
+def is_cuda():
+    return triton.runtime.driver.active.get_current_target().backend == "cuda"
+
+
+def supports_pdl():
+    return is_cuda() and torch.cuda.get_device_capability()[0] >= 9
+
+
+# In this example
+@triton.jit
+def add_kernel(x_ptr,  #
+               y_ptr,  #
+               output_ptr,  #
+               n_elements,  #
+               BLOCK_SIZE: tl.constexpr,  #
+               USE_GDC: tl.constexpr,  #
+               ):
+    pid = tl.program_id(axis=0)
+    block_start = pid * BLOCK_SIZE
+    offsets = block_start + tl.arange(0, BLOCK_SIZE)
+    mask = offsets < n_elements
+    if USE_GDC:
+        # GDC wait waits for ALL programs in the the prior kernel to complete before continuing.
+        # This ensures any memory operations happen before the wait in program order,
+        # e.g. if the prior kernel writes to x or y the new values will be visible.
+        tl.extra.cuda.gdc_wait()
+
+    x = tl.load(x_ptr + offsets, mask=mask)
+    y = tl.load(y_ptr + offsets, mask=mask)
+    if USE_GDC:
+        # GDC launch dependents hints the runtime system to launch dependent kernels.
+        # These dependent kernels must also be launched with PDL enabled.
+        # Once GDC launch has been issued by at least one thread on ALL programs or
path: python/tutorials/11-programmatic-dependent-launch.py
user: login: depaulmillz
body: Looks like it wasn't staged in the last push. Just updated it.
created_at: 2025-04-29T15:43:31Z
author_association: CONTRIBUTOR

PR comments:
user: login: Jokeren
created_at: 2025-04-04T00:43:21Z
body: > Unrelated to this PR, but why can't we also have an nvgpu pass pessimistically insert gdc_wait() and gdc_launcu_dependents() at the start and end of each kernel if they were not added by a user? Most kernels have some amount of code before the first access to global memory.

I'm down for this API-based approach. I'm not sure if it offers benefits to all the kernels or just some of them, and if it enforces any restrictions in creating cuda graphs. 
author_association: CONTRIBUTOR
user: login: binarybana
created_at: 2025-04-04T16:01:10Z
body: > Unrelated to this PR, but why can't we also have an nvgpu pass pessimistically insert gdc_wait() and gdc_launcu_dependents() at the start and end of each kernel if they were not added by a user? Most kernels have some amount of code before the first access to global memory.

That is the plan to do this as a follow-up, but we wanted to introduce the core mechanisms first and show the performance advantages that motivate the conservative (aka pessimistic) compiler added version.
author_association: CONTRIBUTOR