Setting allow_fp16_reduced_precision_reduction via libtorch

elvircrn: <p>The documentation <a href="https://pytorch.org/docs/stable/notes/cuda.html#reduced-precision-reduction-in-fp16-gemms" class="inline-onebox" rel="noopener nofollow ugc">CUDA semantics — PyTorch 2.5 documentation</a></p>
<p>implies that in order to set allow_fp16_reduced_precision_reduction to false via C++/libtorch, one must use setAllowFP16ReductionCuBLAS.</p>
<p>Since calling matmul invokes cutlass, does this mean that disabling fp16 reduction is simply not possible via the C++ API?</p>

ptrblck: <p>The attribute is changing the cuBLAS behavior. Could you describe how exactly you are calling matmuls in CUTLASS?</p>

elvircrn: <p>If I run:</p>
<pre><code class="lang-auto">import torch

if __name__ == '__main__':
    k = 2
    device = 'cuda'
    for m in [4096]:
        for n in [4096]:
            d = torch.zeros((m, n), dtype=torch.float16, device=device)
            x = torch.zeros((n, k), dtype=torch.float16, device=device)
            y = torch.zeros((m, k), dtype=x.dtype, device=x.device).contiguous()
            torch.matmul(d, x, out=y)
</code></pre>
<p>under the profiler</p>
<p>ncu python3 demo.py</p>
<p>I can see that the underlying kernel that gets called is:</p>
<p>cutlass_80_wmma_tensorop_f16_s161616gemm_f16_32x32_128x1_nn_align2</p>
<p>If I modify demo.py in the following way:</p>
<pre><code class="lang-auto">import torch

torch.backends.cuda.matmul.allow_fp16_reduced_precision_reduction = False

if __name__ == '__main__':
    k = 2
    device = 'cuda'
    for m in [4096]:
        for n in [4096]:
            d = torch.zeros((m, n), dtype=torch.float16, device=device)
            x = torch.zeros((n, k), dtype=torch.float16, device=device)
            y = torch.zeros((m, k), dtype=x.dtype, device=x.device).contiguous()
            torch.matmul(d, x, out=y)
</code></pre>
<p>and run the profiler again</p>
<p>cutlass_80_wmma_tensorop_f16_s161616gemm_f16_32x32_128x1_nn_align2</p>
<p>gets called, again.</p>
<p>However, the documentation seems to imply (I could be wrong on this), that torch.backends.cuda.matmul.allow_fp16_reduced_precision_reduction = False and setAllowFP16ReductionCuBLAS(false) are equivalent, which is the source of my confusion.</p>
<p>Is perhaps libtorch missing setAllowFP16ReductionCutlass()?</p>

elvircrn: <p>Shouldn’t these flags affect the underlying kernel being called?</p>

ptrblck: <p>These CUTLASS kernels are called via cuBLAS. Why do you think this flag is not applied?</p>

elvircrn: <aside class="quote no-group" data-username="elvircrn" data-post="3" data-topic="213951">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/elvircrn/48/74418_2.png" class="avatar"> elvircrn:</div>
<blockquote>
<p>cutlass_80_wmma_tensorop_f16_s161616gemm_f16_32x32_128x1_nn_align2</p>
</blockquote>
</aside>
<p>If I pass the --accum flag to be either f32 or f16 to the cutlass profiler,</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/NVIDIA/cutlass/blob/main/media/docs/profiler.md#example-tensor-core-gemm-operations">
  <header class="source">

      <a href="https://github.com/NVIDIA/cutlass/blob/main/media/docs/profiler.md#example-tensor-core-gemm-operations" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/NVIDIA/cutlass/blob/main/media/docs/profiler.md#example-tensor-core-gemm-operations" target="_blank" rel="noopener nofollow ugc">NVIDIA/cutlass/blob/main/media/docs/profiler.md#example-tensor-core-gemm-operations</a></h4>


      <pre><code class="lang-md">![ALT](../images/gemm-hierarchy-with-epilogue-no-labels.png "CUTLASS Profiler")

[README](../../README.md#documentation) &gt; **CUTLASS Profiler**

# CUTLASS Profiler

The CUTLASS Profiler is a command-line driven test and profiling environment for CUTLASS computations
defined in the CUTLASS Instance Library. The CUTLASS Profiler is capable of executing each GEMM, Sparse Gemm,
Conv2d, and Conv3d kernel.

The CUTLASS Profiler may be compiled with:
```bash
$ make cutlass_profiler -j
```

To limit compilation time, only one tile size (typically 128x128) and threadblock cluster size (typically 2x1x1) is instantiated for each data type,
math instruction, and layout. To instantiate all sizes, set the following environment variable when running CMake from an
empty `build/` directory.
```bash
$ cmake .. -DCUTLASS_NVCC_ARCHS="70;75;80" -DCUTLASS_LIBRARY_KERNELS=all  -DCUTLASS_UNITY_BUILD_ENABLED=ON
</code></pre>



  This file has been truncated. <a href="https://github.com/NVIDIA/cutlass/blob/main/media/docs/profiler.md#example-tensor-core-gemm-operations" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>the underlying kernel that gets called also changes (the difference is usually either a lack or inclusion of “_f16”).</p>
<p>I may just be horribly wrong on this, I just assumed that I would also be able to observe something similar in torch matmul.</p>