Package: nvidia-cutlass
CUTLASS 3.4CUTLASS 3.4 - February 2024CUTLASS is a collection of CUDA C++ template abstractions for implementing
high-performance matrix-matrix multiplication (GEMM) and related computations at all levels
and scales within CUDA. It incorporates strategies for hierarchical decomposition and
data movement similar to those used to implement cuBLAS and cuDNN.  CUTLASS decomposes
these "moving parts" into reusable, modular software components abstracted by C++ template
classes.  Primitives for different levels of a conceptual parallelization hierarchy
can be specialized and tuned via custom tiling sizes, data types,
and other algorithmic policy. The resulting flexibility simplifies their use
as building blocks within custom kernels and applications.To support a wide variety of applications, CUTLASS provides extensive support for
mixed-precision computations, providing specialized data-movement and
multiply-accumulate abstractions for half-precision floating
point (FP16), BFloat16 (BF16), Tensor Float 32 (TF32),
single-precision floating point (FP32),FP32 emulation via tensor core instruction,
double-precision floating
point (FP64) types, integer data types (4b and 8b), and binary data types (1b).
CUTLASS demonstrates warp-synchronous matrix multiply operations
targeting the programmable, high-throughputTensor Coresimplemented by
NVIDIA's Volta, Turing, Ampere, and Hopper architectures.See theQuick Start Guideto get started quickly.See thefunctionality listingfor the list of operations
supported at each level of the execution model hierarchy.CUTLASS 3.0 introduced a new core library, CuTe, to describe and manipulate tensors of threads and data.
CuTe is a collection of C++ CUDA template abstractions for defining and operating on hierarchically multidimensional layouts of threads and data. CuTe providesLayoutandTensorobjects that compactly package the type, shape, memory space, and layout of data, while performing the complicated indexing for the user. This lets programmers focus on the logical descriptions of their algorithms while CuTe does the mechanical bookkeeping for them. With these tools, we can quickly design, implement, and modify all dense linear algebra operations.The core abstractions of CuTe are hierarchically multidimensional layouts which can be composed with data arrays to represent tensors. The representation of layouts is powerful enough to represent nearly everything we need to implement efficient dense linear algebra. Layouts can also be combined and manipulated via functional composition, on which we build a large set of common operations such as tiling and partitioning.CUTLASS 3.0 and beyond adopts CuTe throughout the GEMM hierarchy in its templates. This greatly simplifies the design
and improves code composability and readability. More documentation specific to CuTe can be found in itsdedicated documentation directory.In addition to GEMMs, CUTLASS implements high-performance convolution via the implicit GEMM algorithm. Implicit GEMM is the formulation of a convolution operation as a GEMM thereby taking advantage of CUTLASS's modular GEMM pipeline. This allows CUTLASS to build convolutions by reusing highly-optimized GEMM components.What's New in CUTLASS 3.4CUTLASS 3.4.1 is an update to CUTLASS adding:Statically availableCUTLASS Version macrosthat allow for handling API changes between CUTLASS releases on the users' side.Improvements for HopperGroup-GEMMandPointer-Array Batched GEMM.Updates and bugfixes from the community (thanks!).CUTLASS 3.4.0 is an update to CUTLASS adding:ImprovedMixed-input Hopper GEMMssupporting {16-bit, 8-bit} x {8-bit, 4-bit} input types with fast numerical converters and group scaling factors tuned for optimal performance on Hopper H100.Beta release ofPointer-Array Batched GEMMsutilizing TMA and Hopper H100 tensor cores now available. (Requires CUDA 12.3 or above)Beta release ofGroup-GEMM- commonly used in optimization of Mixture-Of-Expert models, is now available on Hopper GPUs taking advantage of TMA and Hopper H100 tensor cores. (Requires CUDA 12.3 or above)Ampere Sparse GEMMsupports Epilogue Visitor Tree (EVT) now.Improvements to NamedBarriers including details ofReservedNamedBarriersused within the CUTLASS library.ImprovedCuTe documentationincluding improved clarity and depth ofQuickstart,CuTe Layout, andCuTe Layout Algebra. Associated code comments, post-conditions, and details inCuTe Core Unit Testsalso improved.Minimum requirements:Architecture: VoltaCompiler: Must support at least C++17CUDA Toolkit version: 11.4Starting from CUTLASS 3.0, CUTLASS removed support for the following:Maxwell and Pascal GPU architecturesUbuntu 16.04CUDA 10.2C++ language versions less than 17.See theCHANGELOGfor a detailed listing of releases and updates.PerformanceCUTLASS primitives are very efficient.  When used to construct device-wide GEMM kernels,
they exhibit peak performance comparable to cuBLAS for scalar GEMM
computations. The above figure shows CUTLASS performance relative to cuBLAS
for large matrix dimensions on anNVIDIA H100(NVIDIA Hopper architecture),
anNVIDIA L40(NVIDIA Ada architecture),
anNVIDIA A100(NVIDIA Ampere architecture),and anNVIDIA A40(NVIDIA Ampere architecture).
CUTLASS 3.0 was compiled with theCUDA 12.0 Toolkit.
Tensor Core operations are implemented using CUDA'smmaandwgmmainstructions.When using CUTLASS building blocks to construct device-wide implicit gemm (Fprop, Dgrad, and Wgrad)
kernels, CUTLASS performance is also comparable to cuDNN when running Resnet-50 layers on anNVIDIA A100as shown in the above figure.  Tensor Core operations are implemented using CUDA'smma instruction.CompatibilityCUTLASS requires a C++17 host compiler and
performs best when built with theCUDA 12.3.2 Toolkit.
It is also compatible with CUDA 11.4, CUDA 11.5, CUDA 11.6, CUDA 11.7, CUDA 11.8, CUDA 12.0, CUDA 12.1, CUDA 12.2.2, CUDA 12.3.1 and CUDA 12.3.2.Operating SystemsWe have tested the following environments.Operating SystemCompilerUbuntu 18.04GCC 7.5.0Ubuntu 20.04GCC 10.3.0Ubuntu 22.04GCC 11.2.0Ubuntu 22.04Clang 10.0.0Ubuntu 22.04Clang 14.0.6Ubuntu 22.04Clang 17.0.6Windows 10.0Visual Studio 2019 v16.11.27Note: GCC 8.5.0 has known regressions regarding fold expressions and overloaded operators. Using GCC 7.5.0 or (preferred) GCC >= 9 is recommended.HardwareCUTLASS runs successfully on the following NVIDIA GPUs, and it is expected to be efficient on Volta, Turing, Ampere, Ada, and Hopper architecture based NVIDIA GPUs.GPUCUDA Compute CapabilityMinimum CUDA Toolkit Required by CUTLASS-3NVIDIA V100 Tensor Core GPU7.011.4NVIDIA TitanV7.011.4NVIDIA GeForce RTX 2080 TI, 2080, 20707.511.4NVIDIA T47.511.4NVIDIA A100 Tensor Core GPU8.011.4NVIDIA A108.611.4NVIDIA GeForce RTX 30908.611.4NVIDIA GeForce RTX 40908.911.8NVIDIA L408.911.8NVIDIA H100 Tensor Core GPU9.011.8Target ArchitectureIn general, PTX code generated for one target architecture can be run on future architectures (i.e., it is forward compatible).  However, CUDA 12.0 introduced the concept of "architecture-accelerated features" whose PTX does not have forward compatibility guarantees. Several Hopper PTX instructions fall under this category of architecture-accelerated features, and thus require asm_90atarget architecture (note the "a" appended). For more details on this and other architecture-accelerated instructions, please refer to theCUDA Documentation.The target architecture information is passed on to CUTLASS via the cmake flagCUTLASS_NVCC_ARCHS. In order to maximize performance on Hopper GH100, users are required to build CUTLASS with90aas the target architecture. If a user accidentally builds a kernel which uses SM90a features (e.g. Hopper Tensor Core Instructions), using the SM90 target (note the lack of "a"), with either CTK 12 or 11.8, the kernel is expected to fail with a runtime error.cmake .. -DCUTLASS_NVCC_ARCHS="90a"Please refer to thefunctionality documentationfor details on which kernels require which target architectures.DocumentationCUTLASS is described in the following documents and the accompanyingDoxygen documentation.Quick Start Guide- build and run CUTLASSFunctionality- summarizes functionality available in CUTLASSEfficient GEMM in CUDA- describes how GEMM kernels may be implemented efficiently in CUDACUTLASS 3.x Design- describes the CUTLASS 3.x design, its benefits, and how CuTe enables us to write much more composable componentsGEMM API 3.x- describes the CUTLASS 3.x GEMM model and C++ template conceptsGEMM API 2.x- describes the CUTLASS 2.x GEMM model and C++ template conceptsImplicit GEMM Convolution- describes 2-D and 3-D convolution in CUTLASSCode Organization- describes the organization and contents of the CUTLASS projectTerminology- describes terms used in the codeProgramming Guidelines- guidelines for writing efficient modern CUDA C++Fundamental types- describes basic C++ classes used in CUTLASS to represent numeric quantities and arraysLayouts- describes layouts of matrices and tensors in memoryTile Iterators- describes C++ concepts for iterating over tiles of matrices in memoryCUTLASS Profiler- command-line driven profiling applicationCUTLASS Utilities- additional templates used to facilate rapid developmentResourcesWe have also described the structure of an efficient GEMM in our talk at theGPU Technology Conference 2018.CUTLASS: Software Primitives for Dense Linear Algebra at All Levels and Scales within CUDADeveloping CUDA Kernels to Push Tensor Cores to the Absolute Limit on NVIDIA A100Accelerating Convolution with Tensor Cores in CUTLASSAccelerating Backward Data Gradient by Increasing Tensor Core Utilization in CUTLASSCUTLASS: Python API, Enhancements, and NVIDIA HopperBuilding CUTLASSCUTLASS is a header-only template library and does not need to be built to be used by other
projects. Client applications should target CUTLASS'sinclude/directory in their include
paths.CUTLASS unit tests, examples, and utilities can be build with CMake.
The minimum version of CMake is given in theQuickstart guide.
Make sure theCUDACXXenvironment  variable points to NVCC in the CUDA Toolkit installed
on your system.$exportCUDACXX=${CUDA_INSTALL_PATH}/bin/nvccCreate a build directory within the CUTLASS project, then run CMake. By default CUTLASS will build kernels
for CUDA architecture versions 5.0, 6.0, 6.1, 7.0, 7.5, 8.0, 8.6, 8.9, and 9.0.
To reduce compile time you can specify
the architectures to build CUTLASS for by changing the CMake configuration settingCUTLASS_NVCC_ARCHS.$mkdirbuild&&cdbuild

$cmake..-DCUTLASS_NVCC_ARCHS=80# compiles for NVIDIA's Ampere ArchitectureFrom thebuild/directory, compile and run the CUTLASS unit tests by building the targettest_unitwith make.The unit tests are organized as several binaries mirroring the top-level namespaces of CUTLASS,
and they may be executed in parallel via make's-jcommand line argument.$maketest_unit-j
...
...
...[----------]Globaltestenvironmenttear-down[==========]946testsfrom57testcasesran.(10812mstotal)[PASSED]946tests.All tests should pass on supported platforms, though the exact number of tests may vary over time.Project StructureCUTLASS is arranged as a header-only library along with Utilities, Tools, Examples, and unit tests.Doxygen documentationprovides a complete list of files, classes,
and template concepts defined in the CUTLASS project.A detailed explanation of the source code organization may be found in theCUTLASS documentation, but several main components are summarized below.CUTLASS Template Libraryinclude/                     # client applications should target this directory in their build's include paths

  cutlass/                   # CUDA Templates for Linear Algebra Subroutines and Solvers - headers only

    arch/                    # direct exposure of architecture features (including instruction-level GEMMs)

    conv/                    # code specialized for convolution

    epilogue/                # code specialized for the epilogue of gemm/convolution

    gemm/                    # code specialized for general matrix product computations

    layout/                  # layout definitions for matrices, tensors, and other mathematical objects in memory

    platform/                # CUDA-capable Standard Library components

    reduction/               # bandwidth-limited reduction kernels that do not fit the "gemm" model

    thread/                  # simt code that can be performed within a CUDA thread
    
    transform/               # code specialized for layout, type, and domain transformations

    *                        # core vocabulary types, containers, and basic numeric operations

  cute/                      # CuTe Layout, layout algebra, MMA/Copy atoms, tiled MMA/Copy

    algorithm/               # Definitions of core operations such as copy, gemm, and operations on cute::tuples

    arch/                    # Bare bones PTX wrapper structs for copy and math instructions

    atom/                    # Meta-information either link to or built from arch/ operators

      mma_atom.hpp           # cute::Mma_Atom and cute::TiledMma

      copy_atom.hpp          # cute::Copy_Atom and cute::TiledCopy

      *sm*.hpp               # Arch specific meta-information for copy and math operations

    *                        # Core library types such as Shape, Stride, Layout, Tensor, and associated operationsCUTLASS SDK ExamplesCUTLASS SDK examplesapply CUTLASS templates to implement basic computations.Toolstools/
  library/                   # CUTLASS Instance Library - contains instantiations of all supported CUTLASS templates
    include/
      cutlass/
        library/

  profiler/                  # CUTLASS Profiler         - command-line utility for executing operations in the
                             #                            CUTLASS Library
  
  util/                      # CUTLASS Utilities        - contains numerous helper classes for
    include/                 #                            manging tensors in device memory, reference
      cutlass/               #                            implementations for GEMM, random initialization
        util/                #                            of tensors, and I/O.TestThetest/unit/directory consist of unit tests implemented with Google Test that demonstrate
basic usage of Core API components and complete tests of the CUTLASS GEMM computations.Instructions for building and running the Unit tests are described in theQuickstart guide.Performance ProfilingThetools/profiler/directory contains a command-line utility for launching each of the GEMM kernels.
It can be built as follows:$makecutlass_profiler-j16Building all GEMM and Convolution kernels (longbuild times)By default, only one tile size is instantiated for each data type, math instruction, and layout.
To instantiate all, set the following environment variable when running CMake from an emptybuild/directory.
Beware, this results intens of thousandsof kernels and long build times.
This would also result in a large binary size and on some platforms linker to fail on building the library.
Therefore, it's highly recommended to generate only a subset of kernels as demonstrated in the sub-section below.$cmake..-DCUTLASS_NVCC_ARCHS=90a-DCUTLASS_LIBRARY_KERNELS=all
...
$makecutlass_profiler-j16Building a subset of GEMM and Convolution kernels (reducedbuild times)To compile strictly one kernel or a small set of kernels, a comma-delimited list of kernel names with
wildcard characters may be used to reduce the set of kernels. The following examples show building exactly one
or a subset of kernels for NVIDIA Ampere and Turing architecture:Building a subset Tensor Core GEMM kernelsTo compile a subset of Tensor Core GEMM kernels with FP32 accumulation and FP16 input targeting NVIDIA Ampere and Turing architecture,
use the below cmake command line:$cmake..-DCUTLASS_NVCC_ARCHS='75;80'-DCUTLASS_LIBRARY_KERNELS=cutlass_tensorop_s*gemm_f16_*_nt_align8
...
$makecutlass_profiler-j16Example command line for profiling a subset of Tensor Core GEMM kernels is as follows:./tools/profiler/cutlass_profiler--kernels=cutlass_tensorop_s*gemm_f16_*_nt_align8--m=3456--n=4096--k=4096...=============================ProblemID:1Provider:CUTLASSOperationKind:gemmOperation:cutlass_tensorop_s1688gemm_f16_256x128_32x2_nt_align8Status:SuccessVerification:ONDisposition:Passed

reference_device:PassedcuBLAS:PassedArguments:--gemm_kind=universal--m=3456--n=4096--k=4096--A=f16:column--B=f16:row--C=f32:column--alpha=1\--beta=0--split_k_slices=1--batch_count=1--op_class=tensorop--accum=f32--cta_m=256--cta_n=128\--cta_k=32--stages=2--warps_m=4--warps_n=2--warps_k=1--inst_m=16--inst_n=8--inst_k=8--min_cc=75\--max_cc=1024Bytes:118489088bytesFLOPs:115992428544flopsRuntime:1.55948msMemory:70.7616GiB/sMath:74378.8GFLOP/s=============================...Building one CUDA Core GEMM kernelTo compile one SGEMM kernel targeting NVIDIA Ampere and Turing architecture, use the below cmake command line:$cmake..-DCUTLASS_NVCC_ARCHS='75;80'-DCUTLASS_LIBRARY_KERNELS=cutlass_simt_sgemm_128x128_8x2_nn_align1
...
$makecutlass_profiler-j16Example command line for profiling single SGEMM CUDA kernel is as follows:$./tools/profiler/cutlass_profiler--kernels=sgemm--m=3456--n=4096--k=4096=============================ProblemID:1Provider:CUTLASSOperationKind:gemmOperation:cutlass_simt_sgemm_128x128_8x2_nn_align1Status:SuccessVerification:ONDisposition:PassedcuBLAS:PassedArguments:--m=3456--n=4096--k=4096--A=f32:column--B=f32:column--C=f32:column--alpha=1--beta=0--split_k_slices=1\--batch_count=1--op_class=simt--accum=f32--cta_m=128--cta_n=128--cta_k=8--stages=2--warps_m=4\--warps_n=2--warps_k=1--inst_m=1--inst_n=1--inst_k=1--min_cc=50--max_cc=1024Bytes:180355072bytesFLOPs:115992428544flopsRuntime:6.73655msMemory:24.934GiB/sMath:17218.4GFLOP/s=============================Building a subset of Tensor Core Convolution kernelsTo compile a subset of Tensor core convolution kernels implementing forward propagation (fprop) with FP32 accumulation
and FP16 input targeting NVIDIA Ampere and Turing architecture, use the below cmake command line:$cmake..-DCUTLASS_NVCC_ARCHS='75;80'-DCUTLASS_LIBRARY_KERNELS=cutlass_tensorop_s*fprop_optimized_f16
...
$makecutlass_profiler-j16Example command line for profiling a subset of Tensor Core convolution kernels is as follows:$./tools/profiler/cutlass_profiler--kernels=cutlass_tensorop_s*fprop_optimized_f16--n=8--h=224--w=224--c=128--k=128--r=3--s=3...=============================ProblemID:1Provider:CUTLASSOperationKind:conv2dOperation:cutlass_tensorop_s16816fprop_optimized_f16_128x128_32x5_nhwcStatus:SuccessVerification:ONDisposition:Passed

reference_device:PassedArguments:--conv_kind=fprop--n=8--h=224--w=224--c=128--k=128--r=3--s=3--p=224--q=224--pad_h=1--pad_w=1\--stride_h=1--stride_w=1--dilation_h=1--dilation_w=1--Activation=f16:nhwc--Filter=f16:nhwc--Output=f32:nhwc\--conv_mode=cross--iterator_algorithm=optimized--alpha=1--beta=0--split_k_mode=serial--split_k_slices=1\--eq_gemm_provider=none--op_class=tensorop--accum=f32--cta_m=128--cta_n=128--cta_k=32--stages=5\--warps_m=2--warps_n=2--warps_k=1--inst_m=16--inst_n=8--inst_k=16--min_cc=80--max_cc=1024Bytes:1130659840bytesFLOPs:118482796544flopsRuntime:0.711496msMemory:1479.99GiB/sMath:166526GFLOP/s=============================...Building one Convolution CUDA kernelTo compile and run one CUDA Core convolution kernel implementing forward propagation (fprop) with F32 accumulation
and FP32 input targeting NVIDIA Ampere and Turing architecture, use the below cmake command line:$cmake..-DCUTLASS_NVCC_ARCHS='75;80'-DCUTLASS_LIBRARY_KERNELS=cutlass_simt_sfprop_optimized_128x128_8x2_nhwc
...
$makecutlass_profiler-j16Example command line for profiling one CUDA Core convolution kernel:$./tools/profiler/cutlass_profiler--kernels=cutlass_simt_sfprop_optimized_128x128_8x2_nhwc--n=8--h=224--w=224--c=128--k=128--r=3--s=3=============================ProblemID:1Provider:CUTLASSOperationKind:conv2dOperation:cutlass_simt_sfprop_optimized_128x128_8x2_nhwcStatus:SuccessVerification:ONDisposition:Passed

reference_device:PassedArguments:--conv_kind=fprop--n=8--h=224--w=224--c=128--k=128--r=3--s=3--p=224--q=224--pad_h=1--pad_w=1\--stride_h=1--stride_w=1--dilation_h=1--dilation_w=1--Activation=f32:nhwc--Filter=f32:nhwc--Output=f32:nhwc\--conv_mode=cross--iterator_algorithm=optimized--alpha=1--beta=0--split_k_mode=serial--split_k_slices=1\--eq_gemm_provider=none--op_class=simt--accum=f32--cta_m=128--cta_n=128--cta_k=8--stages=2--warps_m=4\--warps_n=2--warps_k=1--inst_m=1--inst_n=1--inst_k=1--min_cc=50--max_cc=1024Bytes:2055798784bytesFLOPs:118482796544flopsRuntime:7.34266msMemory:260.752GiB/sMath:16136.2GFLOP/s=============================More Details on Compiling CUTLASS Kernels and CUTLASS ProfilerPlease follow the links for more CMake examples on selectively compiling CUTLASS kernels:GEMM CMake ExamplesImplicit GEMM convolution CMake ExamplesFurther details about the CUTLASS Profiler are described here.AboutCUTLASS is released by NVIDIA Corporation as Open Source software under the3-clause "New" BSD license.ContributorsThe official list of CUTLASS developers and contributors is available here:CONTRIBUTORS.CopyrightCopyright (c) 2017 - 2024 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
SPDX-License-Identifier: BSD-3-ClauseRedistribution and use in source and binary forms, with or without
  modification, are permitted provided that the following conditions are met:

  1. Redistributions of source code must retain the above copyright notice, this
  list of conditions and the following disclaimer.

  2. Redistributions in binary form must reproduce the above copyright notice,
  this list of conditions and the following disclaimer in the documentation
  and/or other materials provided with the distribution.

  3. Neither the name of the copyright holder nor the names of its
  contributors may be used to endorse or promote products derived from
  this software without specific prior written permission.

  THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
  AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
  IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
  DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
  FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
  DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
  SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
  CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
  OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
  OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.