Repository: vllm-project/vllm
Pull request #14245: dynamic distpatch of fp8 kernels
Author: login: jeffdaily
Labels: name: rocm
description: Related to AMD ROCm
name: ready
description: ONLY add when PR is ready to merge/full CI is needed
name: v1
Created: 2025-03-05T00:12:21Z | Merged: 2025-03-11T14:54:56Z

This mostly affects ROCm with hardware that can support one or the other FP8 type. All fp8 kernels are now templated on fp8_type instead of assuming a single fp8_type via `using FP8_TYPE = `. CUDA is largely unaffected.  For CUDA, fp8 kernels only instantiate the OCP type; no binary bloat. For ROCm, two kernel templates are instantiated, one for each fp8 type.

- FP8_TYPE is removed.
- All fp8 kernels have additional fp8_type template param.
- The FP8_E4M3_MAX is replaced with templated struct helper fp8_e4m3_adjusted_max_v.
- is_fp8_ocp() C++ function for host runtime query of preferred fp8 type.
- fp8 kernel launches for ROCm get both OCP and FNUZ templated variants.
- add python APIs current_platform.supports_fp8(), is_fp8_fnuz(), and fp8_dtype().
- add VLLM_DISPATCH_FP8_TYPES that can nest with other dispatch macros


Diff:
diff --git a/benchmarks/kernels/benchmark_moe.py b/benchmarks/kernels/benchmark_moe.py
index 9de8d5af6242..233fc35d2cf5 100644
--- a/benchmarks/kernels/benchmark_moe.py
+++ b/benchmarks/kernels/benchmark_moe.py
@@ -18,8 +18,7 @@
 from vllm.platforms import current_platform
 from vllm.utils import FlexibleArgumentParser
 
-FP8_DTYPE = torch.float8_e4m3fnuz if current_platform.is_rocm(
-) else torch.float8_e4m3fn
+FP8_DTYPE = current_platform.fp8_dtype()
 
 
 class BenchmarkConfig(TypedDict):
diff --git a/csrc/dispatch_utils.h b/csrc/dispatch_utils.h
index 03414b7e1ae9..dc6e0769b878 100644
--- a/csrc/dispatch_utils.h
+++ b/csrc/dispatch_utils.h
@@ -6,6 +6,11 @@
 
 #include <torch/all.h>
 
+// Need a special dispatch case macro since we will nest the FP8 dispatch.
+// Instead of the usual 'scalar_t', this names the dispatched type 'fp8_t'.
+#define AT_DISPATCH_FP8_CASE(enum_type, ...) \
+  AT_PRIVATE_CASE_TYPE_USING_HINT(enum_type, fp8_t, __VA_ARGS__)
+
 #define VLLM_DISPATCH_CASE_FLOATING_TYPES(...)         \
   AT_DISPATCH_CASE(at::ScalarType::Float, __VA_ARGS__) \
   AT_DISPATCH_CASE(at::ScalarType::Half, __VA_ARGS__)  \
@@ -14,17 +19,32 @@
 #define VLLM_DISPATCH_FLOATING_TYPES(TYPE, NAME, ...) \
   AT_DISPATCH_SWITCH(TYPE, NAME, VLLM_DISPATCH_CASE_FLOATING_TYPES(__VA_ARGS__))
 
-// TODO(luka/varun): use FP8_TYPE macro after refactoring
-#ifndef USE_ROCM
-  #define VLLM_DISPATCH_CASE_QUANT_TYPES(...)                    \
-    AT_DISPATCH_CASE(at::ScalarType::Float8_e4m3fn, __VA_ARGS__) \
-    AT_DISPATCH_CASE(at::ScalarType::Char, __VA_ARGS__)
-#else
+// ROCm devices might use either fn or fnuz, so set up dispatch table for both.
+// A host-based check at runtime will create a preferred FP8 type for ROCm
+// such that the correct kernel is dispatched.
+#ifdef USE_ROCM
+  #define VLLM_DISPATCH_CASE_FP8_TYPES(...)                          \
+    AT_DISPATCH_FP8_CASE(at::ScalarType::Float8_e4m3fn, __VA_ARGS__) \
+    AT_DISPATCH_FP8_CASE(at::ScalarType::Float8_e4m3fnuz, __VA_ARGS__)
+
   #define VLLM_DISPATCH_CASE_QUANT_TYPES(...)                      \
+    AT_DISPATCH_CASE(at::ScalarType::Float8_e4m3fn, __VA_ARGS__)   \
     AT_DISPATCH_CASE(at::ScalarType::Float8_e4m3fnuz, __VA_ARGS__) \
     AT_DISPATCH_CASE(at::ScalarType::Char, __VA_ARGS__)
+#else
+  #define VLLM_DISPATCH_CASE_FP8_TYPES(...) \
+    AT_DISPATCH_FP8_CASE(at::ScalarType::Float8_e4m3fn, __VA_ARGS__)
+
+  #define VLLM_DISPATCH_CASE_QUANT_TYPES(...)                    \
+    AT_DISPATCH_CASE(at::ScalarType::Float8_e4m3fn, __VA_ARGS__) \
+    AT_DISPATCH_CASE(at::ScalarType::Char, __VA_ARGS__)
 #endif
 
+// When using this dispatch macro, the type is 'fp8_t' not 'scalar_t'.
+// See AT_DISPATCH_FP8_CASE above.
+#define VLLM_DISPATCH_FP8_TYPES(TYPE, NAME, ...) \
+  AT_DISPATCH_SWITCH(TYPE, NAME, VLLM_DISPATCH_CASE_FP8_TYPES(__VA_ARGS__))
+
 #define VLLM_DISPATCH_QUANT_TYPES(TYPE, NAME, ...) \
   AT_DISPATCH_SWITCH(TYPE, NAME, VLLM_DISPATCH_CASE_QUANT_TYPES(__VA_ARGS__))
 
diff --git a/csrc/layernorm_quant_kernels.cu b/csrc/layernorm_quant_kernels.cu
index c18e2a4e4abe..d595b9e889c8 100644
--- a/csrc/layernorm_quant_kernels.cu
+++ b/csrc/layernorm_quant_kernels.cu
@@ -21,9 +21,9 @@
 namespace vllm {
 
 // TODO(woosuk): Further optimize this kernel.
-template <typename scalar_t>
+template <typename scalar_t, typename fp8_type>
 __global__ void rms_norm_static_fp8_quant_kernel(
-    FP8_TYPE* __restrict__ out,           // [..., hidden_size]
+    fp8_type* __restrict__ out,           // [..., hidden_size]
     const scalar_t* __restrict__ input,   // [..., hidden_size]
     const scalar_t* __restrict__ weight,  // [hidden_size]
     const float* __restrict__ scale,      // [1]
@@ -52,7 +52,7 @@ __global__ void rms_norm_static_fp8_quant_kernel(
     float x = (float)input[blockIdx.x * hidden_size + idx];
     float const out_norm = ((scalar_t)(x * s_variance)) * weight[idx];
     out[blockIdx.x * hidden_size + idx] =
-        scaled_fp8_conversion<true>(out_norm, scale_inv);
+        scaled_fp8_conversion<true, fp8_type>(out_norm, scale_inv);
   }
 }
 
@@ -60,10 +60,10 @@ __global__ void rms_norm_static_fp8_quant_kernel(
    Additional optimizations we can make in this case are
    packed and vectorized operations, which help with the
    memory latency bottleneck. */
-template <typename scalar_t, int width>
+template <typename scalar_t, int width, typename fp8_type>
 __global__ std::enable_if_t<(width > 0) && _typeConvert<scalar_t>::exists>
 fused_add_rms_norm_static_fp8_quant_kernel(
-    FP8_TYPE* __restrict__ out,           // [..., hidden_size]
+    fp8_type* __restrict__ out,           // [..., hidden_size]
     scalar_t* __restrict__ input,         // [..., hidden_size]
     scalar_t* __restrict__ residual,      // [..., hidden_size]
     const scalar_t* __restrict__ weight,  // [hidden_size]
@@ -114,7 +114,7 @@ fused_add_rms_norm_static_fp8_quant_kernel(
 #pragma unroll
     for (int i = 0; i < width; ++i) {
       out[id * width + i] =
-          scaled_fp8_conversion<true>(float(temp.data[i]), scale_inv);
+          scaled_fp8_conversion<true, fp8_type>(float(temp.data[i]), scale_inv);
     }
   }
 }
@@ -122,10 +122,10 @@ fused_add_rms_norm_static_fp8_quant_kernel(
 /* Generic fused_add_rms_norm_kernel
    The width field is not used here but necessary for other specializations.
  */
-template <typename scalar_t, int width>
+template <typename scalar_t, int width, typename fp8_type>
 __global__ std::enable_if_t<(width == 0) || !_typeConvert<scalar_t>::exists>
 fused_add_rms_norm_static_fp8_quant_kernel(
-    FP8_TYPE* __restrict__ out,           // [..., hidden_size]
+    fp8_type* __restrict__ out,           // [..., hidden_size]
     scalar_t* __restrict__ input,         // [..., hidden_size]
     scalar_t* __restrict__ residual,      // [..., hidden_size]
     const scalar_t* __restrict__ weight,  // [hidden_size]
@@ -158,7 +158,7 @@ fused_add_rms_norm_static_fp8_quant_kernel(
     float x = (float)residual[blockIdx.x * hidden_size + idx];
     float const out_norm = ((scalar_t)(x * s_variance)) * weight[idx];
     out[blockIdx.x * hidden_size + idx] =
-        scaled_fp8_conversion<true>(out_norm, scale_inv);
+        scaled_fp8_conversion<true, fp8_type>(out_norm, scale_inv);
   }
 }
 
@@ -176,25 +176,33 @@ void rms_norm_static_fp8_quant(torch::Tensor& out,     // [..., hidden_size]
   dim3 block(std::min(hidden_size, 1024));
   const at::cuda::OptionalCUDAGuard device_guard(device_of(input));
   const cudaStream_t stream = at::cuda::getCurrentCUDAStream();
-  VLLM_DISPATCH_FLOATING_TYPES(input.scalar_type(), "rms_norm_kernel", [&] {
-    vllm::rms_norm_static_fp8_quant_kernel<scalar_t>
-        <<<grid, block, 0, stream>>>(
-            out.data_ptr<FP8_TYPE>(), input.data_ptr<scalar_t>(),
-            weight.data_ptr<scalar_t>(), scale.data_ptr<float>(), epsilon,
-            num_tokens, hidden_size);
-  });
+  VLLM_DISPATCH_FLOATING_TYPES(
+      input.scalar_type(), "rms_norm_kernel_scalar_type", [&] {
+        VLLM_DISPATCH_FP8_TYPES(
+            out.scalar_type(), "rms_norm_kernel_fp8_type", [&] {
+              vllm::rms_norm_static_fp8_quant_kernel<scalar_t, fp8_t>
+                  <<<grid, block, 0, stream>>>(
+                      out.data_ptr<fp8_t>(), input.data_ptr<scalar_t>(),
+                      weight.data_ptr<scalar_t>(), scale.data_ptr<float>(),
+                      epsilon, num_tokens, hidden_size);
+            });
+      });
 }
 
-#define LAUNCH_FUSED_ADD_RMS_NORM(width)                                    \
-  VLLM_DISPATCH_FLOATING_TYPES(                                             \
-      input.scalar_type(), "fused_add_rms_norm_kernel", [&] {               \
-        vllm::fused_add_rms_norm_static_fp8_quant_kernel<scalar_t, width>   \
-            <<<grid, block, 0, stream>>>(                                   \
-                out.data_ptr<FP8_TYPE>(), input.data_ptr<scalar_t>(),       \
-                residual.data_ptr<scalar_t>(), weight.data_ptr<scalar_t>(), \
-                scale.data_ptr<float>(), epsilon, num_tokens, hidden_size); \
+#define LAUNCH_FUSED_ADD_RMS_NORM(width)                                     \
+  VLLM_DISPATCH_FLOATING_TYPES(                                              \
+      input.scalar_type(), "fused_add_rms_norm_kernel_scalar_type", [&] {    \
+        VLLM_DISPATCH_FP8_TYPES(                                             \
+            out.scalar_type(), "fused_add_rms_norm_kernel_fp8_type", [&] {   \
+              vllm::fused_add_rms_norm_static_fp8_quant_kernel<scalar_t,     \
+                                                               width, fp8_t> \
+                  <<<grid, block, 0, stream>>>(                              \
+                      out.data_ptr<fp8_t>(), input.data_ptr<scalar_t>(),     \
+                      residual.data_ptr<scalar_t>(),                         \
+                      weight.data_ptr<scalar_t>(), scale.data_ptr<float>(),  \
+                      epsilon, num_tokens, hidden_size);                     \
+            });                                                              \
       });
-
 void fused_add_rms_norm_static_fp8_quant(
     torch::Tensor& out,       // [..., hidden_size],
     torch::Tensor& input,     // [..., hidden_size]
diff --git a/csrc/quantization/fp8/amd/quant_utils.cuh b/csrc/quantization/fp8/amd/quant_utils.cuh
index b812b28b607e..f01427cc3d0c 100644
--- a/csrc/quantization/fp8/amd/quant_utils.cuh
+++ b/csrc/quantization/fp8/amd/quant_utils.cuh
@@ -13,6 +13,28 @@ namespace vllm {
 namespace fp8 {
   #ifdef ENABLE_FP8
 
+// Use hardware cvt instruction for fp8 on rocm
+template <typename fp8_type>
+__device__ __forceinline__ fp8_type cvt_c10(float const r) {
+  return {};
+}
+
+template <>
+__device__ __forceinline__ c10::Float8_e4m3fn cvt_c10(float const r) {
+  return c10::Float8_e4m3fn(
+      __hip_cvt_float_to_fp8(r, __hip_fp8_e4m3::__default_saturation,
+                             __hip_fp8_e4m3::__default_interpret),
+      c10::Float8_e4m3fn::from_bits());
+}
+
+template <>
+__device__ __forceinline__ c10::Float8_e4m3fnuz cvt_c10(float const r) {
+  return c10::Float8_e4m3fnuz(
+      __hip_cvt_float_to_fp8(r, __hip_fp8_e4m3_fnuz::__default_saturation,
+                             __hip_fp8_e4m3_fnuz::__default_interpret),
+      c10::Float8_e4m3fnuz::from_bits());
+}
+
 template <typename Tout, typename Tin>
 __inline__ __device__ Tout vec_conversion(const Tin& x) {
   return x;
diff --git a/csrc/quantization/fp8/common.cu b/csrc/quantization/fp8/common.cu
index e4f6615ede1e..8f9aa21aae4e 100644
--- a/csrc/quantization/fp8/common.cu
+++ b/csrc/quantization/fp8/common.cu
@@ -11,8 +11,8 @@
 
 namespace vllm {
 
-template <typename scalar_t>
-__global__ void scaled_fp8_quant_kernel(FP8_TYPE* __restrict__ out,
+template <typename scalar_t, typename fp8_type>
+__global__ void scaled_fp8_quant_kernel(fp8_type* __restrict__ out,
                                         const scalar_t* __restrict__ input,
                                         const float* __restrict__ scale,
                                         int64_t num_elems) {
@@ -25,12 +25,13 @@ __global__ void scaled_fp8_quant_kernel(FP8_TYPE* __restrict__ out,
       out, input, inverted_scale, num_elems, tid, blockDim.x * gridDim.x);
 }
 
-template <typename scalar_t>
+template <typename scalar_t, typename fp8_type>
 __global__ void dynamic_per_token_scaled_fp8_quant_kernel(
-    FP8_TYPE* __restrict__ out, float* __restrict__ scale,
+    fp8_type* __restrict__ out, float* __restrict__ scale,
     scalar_t const* __restrict__ input, float const* __restrict__ scale_ub,
     const int hidden_size) {
-  float const min_scaling_factor = 1.0f / (FP8_E4M3_MAX * 512.f);
+  float const min_scaling_factor =
+      1.0f / (fp8_e4m3_adjusted_max_v<fp8_type> * 512.f);
 
   int const tid = threadIdx.x;
   int const token_idx = blockIdx.x;
@@ -38,7 +39,7 @@ __global__ void dynamic_per_token_scaled_fp8_quant_kernel(
   // Use int64 to avoid overflowing an int32 when calculating this offset
   int64_t offset = static_cast<int64_t>(token_idx) * hidden_size;
   scalar_t const* __restrict__ token_input = &input[offset];
-  FP8_TYPE* __restrict__ token_output = &out[offset];
+  fp8_type* __restrict__ token_output = &out[offset];
 
   // For vectorization, token_input and token_output pointers need to be
   // aligned at 8-byte and 4-byte addresses respectively.
@@ -66,7 +67,8 @@ __global__ void dynamic_per_token_scaled_fp8_quant_kernel(
       token_scale = block_absmax_val_maybe;
     }
     // token scale computation
-    token_scale = max(token_scale / FP8_E4M3_MAX, min_scaling_factor);
+    token_scale = max(token_scale / fp8_e4m3_adjusted_max_v<fp8_type>,
+                      min_scaling_factor);
     scale[token_idx] = token_scale;
   }
   __syncthreads();
@@ -77,7 +79,7 @@ __global__ void dynamic_per_token_scaled_fp8_quant_kernel(
         token_output, token_input, token_scale, hidden_size, tid, blockDim.x);
   } else {
     for (int i = tid; i < hidden_size; i += blockDim.x) {
-      token_output[i] = scaled_fp8_conversion<false>(
+      token_output[i] = scaled_fp8_conversion<false, fp8_type>(
           static_cast<float>(token_input[i]), token_scale);
     }
   }
@@ -96,10 +98,14 @@ void static_scaled_fp8_quant(torch::Tensor& out,          // [..., d]
   const at::cuda::OptionalCUDAGuard device_guard(device_of(input));
   const cudaStream_t stream = at::cuda::getCurrentCUDAStream();
   VLLM_DISPATCH_FLOATING_TYPES(
-      input.scalar_type(), "scaled_fp8_quant_kernel", [&] {
-        vllm::scaled_fp8_quant_kernel<scalar_t><<<grid, block, 0, stream>>>(
-            out.data_ptr<FP8_TYPE>(), input.data_ptr<scalar_t>(),
-            scale.data_ptr<float>(), num_elems);
+      input.scalar_type(), "scaled_fp8_quant_kernel_scalar_type", [&] {
+        VLLM_DISPATCH_FP8_TYPES(
+            out.scalar_type(), "scaled_fp8_quant_kernel_fp8_type", [&] {
+              vllm::scaled_fp8_quant_kernel<scalar_t, fp8_t>
+                  <<<grid, block, 0, stream>>>(
+                      out.data_ptr<fp8_t>(), input.data_ptr<scalar_t>(),
+                      scale.data_ptr<float>(), num_elems);
+            });
       });
 }
 
@@ -114,12 +120,18 @@ void dynamic_scaled_fp8_quant(torch::Tensor& out,          // [..., d]
   const at::cuda::OptionalCUDAGuard device_guard(device_of(input));
   const cudaStream_t stream = at::cuda::getCurrentCUDAStream();
   VLLM_DISPATCH_FLOATING_TYPES(
-      input.scalar_type(), "scaled_fp8_quant_kernel", [&] {
-        vllm::segmented_max_reduction<scalar_t><<<grid, block, 0, stream>>>(
-            scale.data_ptr<float>(), input.data_ptr<scalar_t>(), num_elems);
-        vllm::scaled_fp8_quant_kernel<scalar_t><<<grid, block, 0, stream>>>(
-            out.data_ptr<FP8_TYPE>(), input.data_ptr<scalar_t>(),
-            scale.data_ptr<float>(), num_elems);
+      input.scalar_type(), "scaled_fp8_quant_kernel_scalar_type", [&] {
+        VLLM_DISPATCH_FP8_TYPES(
+            out.scalar_type(), "scaled_fp8_quant_kernel_fp8_type", [&] {
+              vllm::segmented_max_reduction<scalar_t, fp8_t>
+                  <<<grid, block, 0, stream>>>(scale.data_ptr<float>(),
+                                               input.data_ptr<scalar_t>(),
+                                               num_elems);
+              vllm::scaled_fp8_quant_kernel<scalar_t, fp8_t>
+                  <<<grid, block, 0, stream>>>(
+                      out.data_ptr<fp8_t>(), input.data_ptr<scalar_t>(),
+                      scale.data_ptr<float>(), num_elems);
+            });
       });
 }
 
@@ -138,12 +150,18 @@ void dynamic_per_token_scaled_fp8_quant(
   const at::cuda::OptionalCUDAGuard device_guard(device_of(input));
   const cudaStream_t stream = at::cuda::getCurrentCUDAStream();
   VLLM_DISPATCH_FLOATING_TYPES(
-      input.scalar_type(), "dynamic_per_token_scaled_fp8_quant_kernel", [&] {
-        vllm::dynamic_per_token_scaled_fp8_quant_kernel<scalar_t>
-            <<<grid, block, 0, stream>>>(
-                out.data_ptr<FP8_TYPE>(), scales.data_ptr<float>(),
-                input.data_ptr<scalar_t>(),
-                scale_ub.has_value() ? scale_ub->data_ptr<float>() : nullptr,
-                hidden_size);
+      input.scalar_type(),
+      "dynamic_per_token_scaled_fp8_quant_kernel_scalar_type", [&] {
+        VLLM_DISPATCH_FP8_TYPES(
+            out.scalar_type(),
+            "dynamic_per_token_scaled_fp8_quant_kernel_fp8_type", [&] {
+              vllm::dynamic_per_token_scaled_fp8_quant_kernel<scalar_t, fp8_t>
+                  <<<grid, block, 0, stream>>>(
+                      out.data_ptr<fp8_t>(), scales.data_ptr<float>(),
+                      input.data_ptr<scalar_t>(),
+                      scale_ub.has_value() ? scale_ub->data_ptr<float>()
+                                           : nullptr,
+                      hidden_size);
+            });
       });
 }
diff --git a/csrc/quantization/fp8/common.cuh b/csrc/quantization/fp8/common.cuh
index fac99b297342..d331c63ae827 100644
--- a/csrc/quantization/fp8/common.cuh
+++ b/csrc/quantization/fp8/common.cuh
@@ -7,18 +7,52 @@
 
 #ifndef USE_ROCM
   #include <c10/util/Float8_e4m3fn.h>
-using FP8_TYPE = c10::Float8_e4m3fn;
-C10_HOST_DEVICE constexpr auto FP8_E4M3_MAX =
-    std::numeric_limits<FP8_TYPE>::max();
+  #define MAYBE_HOST_DEVICE C10_HOST_DEVICE
 #else
+  #include <ATen/hip/HIPContext.h>
+  #include <c10/util/Float8_e4m3fn.h>
   #include <c10/util/Float8_e4m3fnuz.h>
   #include "amd/quant_utils.cuh"
-using FP8_TYPE = c10::Float8_e4m3fnuz;
-// Using the default max value from pytorch (240.0) will cause accuracy
-// issue when running dynamic quantization. Here use 224.0f for rocm.
-constexpr auto FP8_E4M3_MAX = 224.0f;
+  // ROCm doesn't seem to need C10_HOST_DEVICE for static constexpr
+  #define MAYBE_HOST_DEVICE
+#endif
+
+// Determines the preferred FP8 type for the current platform.
+// Note that for CUDA this just returns true,
+// but on ROCm it will check device props.
+static bool is_fp8_ocp() {
+#ifndef USE_ROCM
+  return true;
+#else
+  auto dprops = at::cuda::getCurrentDeviceProperties();
+  std::string device_arch = dprops->gcnArchName;
+  size_t substring = device_arch.find("gfx94");
+  return substring == std::string::npos;
 #endif
-constexpr static auto kFp8Type = c10::CppTypeToScalarType<FP8_TYPE>::value;
+}
+
+template <typename T>
+struct fp8_e4m3_adjusted_max;
+
+template <>
+struct fp8_e4m3_adjusted_max<c10::Float8_e4m3fn> {
+  static constexpr c10::Float8_e4m3fn val() {
+    return std::numeric_limits<c10::Float8_e4m3fn>::max();
+  }
+};
+
+// Using the default max value from pytorch (240.0 0x7F) will cause accuracy
+// issues when running dynamic quantization. Here use 224.0 0x7E for rocm.
+template <>
+struct fp8_e4m3_adjusted_max<c10::Float8_e4m3fnuz> {
+  static constexpr c10::Float8_e4m3fnuz val() {
+    return c10::Float8_e4m3fnuz(0x7E, c10::Float8_e4m3fnuz::from_bits());
+  }
+};
+
+template <typename T>
+MAYBE_HOST_DEVICE static constexpr T fp8_e4m3_adjusted_max_v =
+    fp8_e4m3_adjusted_max<T>::val();
 
 namespace vllm {
 
@@ -32,8 +66,8 @@ __device__ __forceinline__ float atomicMaxFloat(float* addr, float value) {
   return old;
 }
 
-template <bool is_scale_inverted>
-__device__ __forceinline__ FP8_TYPE scaled_fp8_conversion(float const val,
+template <bool is_scale_inverted, typename fp8_type>
+__device__ __forceinline__ fp8_type scaled_fp8_conversion(float const val,
                                                           float const scale) {
   float x = 0.0f;
   if constexpr (is_scale_inverted) {
@@ -42,15 +76,13 @@ __device__ __forceinline__ FP8_TYPE scaled_fp8_conversion(float const val,
     x = val / scale;
   }
 
-  float r = fmax(-FP8_E4M3_MAX, fmin(x, FP8_E4M3_MAX));
+  float r = fmax(-fp8_e4m3_adjusted_max_v<fp8_type>,
+                 fmin(x, fp8_e4m3_adjusted_max_v<fp8_type>));
 #ifndef USE_ROCM
-  return static_cast<c10::Float8_e4m3fn>(r);
+  return static_cast<fp8_type>(r);
 #else
   // Use hardware cvt instruction for fp8 on rocm
-  return c10::Float8_e4m3fnuz(
-      __hip_cvt_float_to_fp8(r, fp8::fp8_type::__default_saturation,
-                             fp8::fp8_type::__default_interpret),
-      c10::Float8_e4m3fnuz::from_bits());
+  return fp8::cvt_c10<fp8_type>(r);
 #endif
 }
 
@@ -60,7 +92,7 @@ __device__ __forceinline__ FP8_TYPE scaled_fp8_conversion(float const val,
 // So to get the right answer, *scale needs to be initialized to
 // a value <= 0.0 and we need to wait for all thread blocks to
 // finish before consuming *scale.
-template <typename scalar_t>
+template <typename scalar_t, typename fp8_type>
 __global__ void segmented_max_reduction(float* __restrict__ scale,
                                         const scalar_t* __restrict__ input,
                                         int64_t num_elems) {
@@ -91,7 +123,7 @@ __global__ void segmented_max_reduction(float* __restrict__ scale,
   // Finally, since cache[0] contains the maximum for this thread block,
   // atomically write the max to the target location
   if (threadIdx.x == 0) {
-    atomicMaxFloat(scale, cache[0] / FP8_E4M3_MAX);
+    atomicMaxFloat(scale, cache[0] / fp8_e4m3_adjusted_max_v<fp8_type>);
   }
 }
 
@@ -123,13 +155,13 @@ __device__ float thread_max_vec(scalar_t const* __restrict__ input,
   return absmax_val;
 }
 
-template <typename scalar_t, bool is_scale_inverted>
-__device__ void scaled_fp8_conversion_vec(FP8_TYPE* __restrict__ out,
+template <typename scalar_t, bool is_scale_inverted, typename fp8_type>
+__device__ void scaled_fp8_conversion_vec(fp8_type* __restrict__ out,
                                           scalar_t const* __restrict__ input,
                                           float const scale,
                                           int64_t const num_elems,
                                           int const tid, int const step) {
-  using float8x4_t = q8x4_t<FP8_TYPE>;
+  using float8x4_t = q8x4_t<fp8_type>;
   // Vectorized input/output to better utilize memory bandwidth.
   auto const* vectorized_in = reinterpret_cast<vec4_t<scalar_t> const*>(input);
   auto* vectorized_out = reinterpret_cast<float8x4_t*>(out);
@@ -141,22 +173,22 @@ __device__ void scaled_fp8_conversion_vec(FP8_TYPE* __restrict__ out,
     vec4_t<scalar_t> in_vec = vectorized_in[i];
     float8x4_t out_vec;
 
-    out_vec.x = scaled_fp8_conversion<is_scale_inverted>(
+    out_vec.x = scaled_fp8_conversion<is_scale_inverted, fp8_type>(
         static_cast<float>(in_vec.x), scale);
-    out_vec.y = scaled_fp8_conversion<is_scale_inverted>(
+    out_vec.y = scaled_fp8_conversion<is_scale_inverted, fp8_type>(
         static_cast<float>(in_vec.y), scale);
-    out_vec.z = scaled_fp8_conversion<is_scale_inverted>(
+    out_vec.z = scaled_fp8_conversion<is_scale_inverted, fp8_type>(
         static_cast<float>(in_vec.z), scale);
-    out_vec.w = scaled_fp8_conversion<is_scale_inverted>(
+    out_vec.w = scaled_fp8_conversion<is_scale_inverted, fp8_type>(
         static_cast<float>(in_vec.w), scale);
     vectorized_out[i] = out_vec;
   }
 
   // Handle the remaining elements if num_elems is not divisible by 4
   for (int64_t i = num_vec_elems * 4 + tid; i < num_elems; i += step) {
-    out[i] = scaled_fp8_conversion<is_scale_inverted>(
+    out[i] = scaled_fp8_conversion<is_scale_inverted, fp8_type>(
         static_cast<float>(input[i]), scale);
   }
 }
 
-}  // namespace vllm
\ No newline at end of file
+}  // namespace vllm
diff --git a/csrc/quantization/fused_kernels/fused_layernorm_dynamic_per_token_quant.cu b/csrc/quantization/fused_kernels/fused_layernorm_dynamic_per_token_quant.cu
index 3c4f183bf4b5..1be89c504bfe 100644
--- a/csrc/quantization/fused_kernels/fused_layernorm_dynamic_per_token_quant.cu
+++ b/csrc/quantization/fused_kernels/fused_layernorm_dynamic_per_token_quant.cu
@@ -144,6 +144,9 @@ void rms_norm_dynamic_per_token_quant(
     torch::Tensor& scales,        // [num_tokens]
     double const var_epsilon,     // Variance epsilon used in norm calculation
     std::optional<at::Tensor> scale_ub, std::optional<at::Tensor> residual) {
+  static c10::ScalarType kFp8Type = is_fp8_ocp()
+                                        ? c10::ScalarType::Float8_e4m3fn
+                                        : c10::ScalarType::Float8_e4m3fnuz;
   TORCH_CHECK(out.dtype() == kFp8Type || out.dtype() == torch::kInt8);
   TORCH_CHECK(out.is_contiguous() && input.is_contiguous());
 
diff --git a/csrc/quantization/fused_kernels/quant_conversions.cuh b/csrc/quantization/fused_kernels/quant_conversions.cuh
index f8a9872226a3..9ac7b188f518 100644
--- a/csrc/quantization/fused_kernels/quant_conversions.cuh
+++ b/csrc/quantization/fused_kernels/quant_conversions.cuh
@@ -31,9 +31,11 @@ static __device__ __forceinline__ int8_t float_to_int8_rn(float const x) {
 #endif
 }
 
-static __device__ __forceinline__ FP8_TYPE float_to_fp8(float const x) {
-  float const r = fmax(-FP8_E4M3_MAX, fmin(x, FP8_E4M3_MAX));
-  return static_cast<FP8_TYPE>(r);
+template <typename fp8_type>
+static __device__ __forceinline__ fp8_type float_to_fp8(float const x) {
+  float const r = fmax(-fp8_e4m3_adjusted_max_v<fp8_type>,
+                       fmin(x, fp8_e4m3_adjusted_max_v<fp8_type>));
+  return static_cast<fp8_type>(r);
 }
 
 template <typename quant_type_t, bool is_scale_inverted, typename enable = void>
@@ -54,15 +56,16 @@ struct ScaledQuant<
 };
 
 template <typename quant_type_t, bool is_scale_inverted>
-struct ScaledQuant<
-    quant_type_t, is_scale_inverted,
-    typename std::enable_if_t<std::is_same_v<quant_type_t, FP8_TYPE>>> {
+struct ScaledQuant<quant_type_t, is_scale_inverted,
+                   typename std::enable_if_t<
+                       std::is_same_v<quant_type_t, c10::Float8_e4m3fn> ||
+                       std::is_same_v<quant_type_t, c10::Float8_e4m3fnuz>>> {
   static __device__ __forceinline__ quant_type_t quant_fn(float const x,
                                                           float const scale) {
     if constexpr (is_scale_inverted) {
-      return float_to_fp8(x * scale);
+      return float_to_fp8<quant_type_t>(x * scale);
     } else {
-      return float_to_fp8(x / scale);
+      return float_to_fp8<quant_type_t>(x / scale);
     }
   }
 };
diff --git a/csrc/quantization/vectorization.cuh b/csrc/quantization/vectorization.cuh
index 44c999130f75..866da10b5bc1 100644
--- a/csrc/quantization/vectorization.cuh
+++ b/csrc/quantization/vectorization.cuh
@@ -4,7 +4,6 @@
  */
 
 // Include both AMD and NVIDIA fp8 types to avoid circular import
-// TODO(luka/varun) use FP8_TYPE instead after refactoring
 #include <c10/util/Float8_e4m3fnuz.h>
 #include <c10/util/Float8_e4m3fn.h>
 
diff --git a/tests/kernels/quant_utils.py b/tests/kernels/quant_utils.py
index a21d642bcaaf..498da6001ae9 100644
--- a/tests/kernels/quant_utils.py
+++ b/tests/kernels/quant_utils.py
@@ -9,8 +9,7 @@
 # Using the default value (240.0) from pytorch will cause accuracy
 # issue on dynamic quantization models. Here use 224.0 for rocm.
 ROCM_FP8_MAX = 224.0
-FP8_DTYPE = torch.float8_e4m3fnuz if current_platform.is_rocm() \
-                else torch.float8_e4m3fn
+FP8_DTYPE = current_platform.fp8_dtype()
 
 
 def as_float32_tensor(x: Union[float, torch.tensor]) -> torch.tensor:
diff --git a/tests/kernels/test_triton_scaled_mm.py b/tests/kernels/test_triton_scaled_mm.py
index bbff3e0a0415..45f10b0eb1d5 100644
--- a/tests/kernels/test_triton_scaled_mm.py
+++ b/tests/kernels/test_triton_scaled_mm.py
@@ -32,11 +32,8 @@ def scaled_mm_torch(a: torch.Tensor,
 
 def get_8bit_types():
     types = [torch.int8]
-    supports_fp8 = current_platform.has_device_capability(89)
-    if current_platform.is_rocm() and supports_fp8:
-        types.append(torch.float8_e4m3fnuz)
-    elif current_platform.is_cuda() and supports_fp8:
-        types.append(torch.float8_e4m3fn)
+    if current_platform.supports_fp8():
+        types.append(current_platform.fp8_dtype())
     return types
 
 
diff --git a/tests/quantization/test_fp8.py b/tests/quantization/test_fp8.py
index 3a7f0a196b5b..b9a1d759b9a4 100644
--- a/tests/quantization/test_fp8.py
+++ b/tests/quantization/test_fp8.py
@@ -103,8 +103,7 @@ def check_model(model):
                 assert attn._v_scale == 1.0
 
             if current_platform.is_cuda():
-                if current_platform.has_device_capability(
-                        89) and not force_marlin:
+                if current_platform.supports_fp8() and not force_marlin:
                     # For GPUs with hardware support, we keep weights in fp8
                     assert fc1.weight.dtype == torch.float8_e4m3fn
                 else:
@@ -112,11 +111,9 @@ def check_model(model):
                     # for weight-only quantization using Marlin kernels
                     assert fc1.weight.dtype == torch.int32
             elif current_platform.is_rocm():
-                # Only MI300 and above support quantization='fp8'
-                if current_platform.has_device_capability(
-                        94) and not force_marlin:
+                if current_platform.supports_fp8() and not force_marlin:
                     # For GPUs with hardware support, we keep weights in fp8
-                    assert fc1.weight.dtype == torch.float8_e4m3fnuz
+                    assert fc1.weight.dtype == current_platform.fp8_dtype()
                 else:  # unsupported ROCm platform
                     pytest.skip(
                         "Skip `test_load_fp16_model`. "
diff --git a/vllm/_custom_ops.py b/vllm/_custom_ops.py
index d25e7944a05a..189e4bde2392 100644
--- a/vllm/_custom_ops.py
+++ b/vllm/_custom_ops.py
@@ -478,16 +478,16 @@ def cutlass_scaled_mm(a: torch.Tensor,
                       out_dtype: torch.dtype,
                       bias: Optional[torch.Tensor] = None) -> torch.Tensor:
     """
-    `cutlass_scaled_mm` implements a fused version of 
+    `cutlass_scaled_mm` implements a fused version of
         `output = torch.mm((scale_a * a), (scale_b * b)).to(out_dtype)`
-    where scale_a * a and scale_b * b are implemented using numpy-style 
-    broadcasting. 
-    
-    In order to support blockwise scaling like found in DeepSeek V3 we also 
-    support extended "group" broadcast rules. We extend the numpy-style 
-    broadcasting rules with the following rule: 
-        "if the extent of a dimension in the source shape is between 1 and 
-        corresponding extent in the target shape we repeat each element along 
+    where scale_a * a and scale_b * b are implemented using numpy-style
+    broadcasting.
+
+    In order to support blockwise scaling like found in DeepSeek V3 we also
+    support extended "group" broadcast rules. We extend the numpy-style
+    broadcasting rules with the following rule:
+        "if the extent of a dimension in the source shape is between 1 and
+        corresponding extent in the target shape we repeat each element along
         that dimension  src_shape[dim] // target_shape[dim] times consecutively"
     example if we have:
           a = [[1, 2], and target_shape = (2, 4)
@@ -564,7 +564,7 @@ def cutlass_sparse_compress(a: torch.Tensor) \
     with Cutlass sparse kernels.
 
     Args:
-        a (torch.Tensor): 
+        a (torch.Tensor):
             The input tensor to be compressed. Must have one of the following data types:
             - `torch.int8`
             - `torch.float8_e4m3fn`
@@ -572,7 +572,7 @@ def cutlass_sparse_compress(a: torch.Tensor) \
             - `torch.float16`
 
     Returns:
-        tuple[torch.Tensor, torch.Tensor]: 
+        tuple[torch.Tensor, torch.Tensor]:
             A tuple containing:
             - `a_nzs` (torch.Tensor): A tensor containing non-zero elements of `a`.
             - `a_meta` (torch.Tensor): A tensor containing metadata for the sparse representation.
@@ -875,9 +875,8 @@ def scaled_fp8_quant(
     # This code assumes batch_dim and num_tokens are flattened
     assert (input.ndim == 2)
     shape: Union[tuple[int, int], torch.Size] = input.shape
-    # For rocm, the output fp8 dtype is torch.float_e3m3fnuz
-    out_dtype: torch.dtype = torch.float8_e4m3fnuz \
-            if current_platform.is_rocm() else torch.float8_e4m3fn
+    # For ROCm on MI300, the output fp8 dtype is torch.float_e3m3fnuz
+    out_dtype: torch.dtype = current_platform.fp8_dtype()
     if num_token_padding:
         shape = (max(num_token_padding, input.shape[0]), shape[1])
     output = torch.empty(shape, device=input.device, dtype=out_dtype)
@@ -908,7 +907,7 @@ def allspark_repack_weight(
         has_zp: bool = False
 ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
     """
-    Rearrange qweight, scale, and zero_point(if asymmetric) to n32k16 format 
+    Rearrange qweight, scale, and zero_point(if asymmetric) to n32k16 format
     for Ampere W8A16 Fused Gemm kernel
 
     Args:
@@ -917,10 +916,10 @@ def allspark_repack_weight(
         zero_point: fp16/bf16 weight zero_point tensor, 1 x n format.
             Must be provided for asymmetric quantization.
         has_zp: if use symmetric quantization, has_zp = False.
-            if use asymmetric quantization, has_zp = True.  
-    
+            if use asymmetric quantization, has_zp = True.
+
     Returns:
-        tuple[torch.Tensor, torch.Tensor, Optional[torch.Tensor]] : 
+        tuple[torch.Tensor, torch.Tensor, Optional[torch.Tensor]] :
             rearranged weight, scale, and optionally zero_point.
     """
     K = qweight.shape[0]
diff --git a/vllm/attention/backends/mla/common.py b/vllm/attention/backends/mla/common.py
index 4f4b70cd8f48..e912b1e9757a 100644
--- a/vllm/attention/backends/mla/common.py
+++ b/vllm/attention/backends/mla/common.py
@@ -226,7 +226,7 @@
     CompressedTensorsW8A8Fp8)
 from vllm.model_executor.layers.quantization.fp8 import Fp8LinearMethod
 from vllm.model_executor.layers.quantization.utils.fp8_utils import (
-    Fp8LinearGenericOp, current_platform_fp8_dtype, is_fp8)
+    Fp8LinearGenericOp, is_fp8)
 from vllm.model_executor.layers.quantization.utils.quant_utils import (
     scaled_quantize)
 from vllm.model_executor.layers.rotary_embedding import (
@@ -1238,7 +1238,7 @@ def get_and_maybe_dequant_weights(layer: LinearBase):
                 W_Q_UK, W_Q_UK_scales = scaled_quantize(
                     W_Q_UK,
                     self.reqaunt_weight_group_shape,
-                    quant_dtype=current_platform_fp8_dtype)
+                    quant_dtype=current_platform.fp8_dtype())
                 # For FP8 save the transpose so we can use
                 # `apply_w8a8_block_fp8_linear` directly
                 self.W_Q_UK = W_Q_UK.T.contiguous()
@@ -1255,7 +1255,7 @@ def get_and_maybe_dequant_weights(layer: LinearBase):
                 W_UV_O, W_UV_O_scales = scaled_quantize(
                     W_UV_O,
                     self.reqaunt_weight_group_shape,
-                    quant_dtype=current_platform_fp8_dtype)
+                    quant_dtype=current_platform.fp8_dtype())
                 # For FP8 save the transpose so we can use
                 # `apply_w8a8_block_fp8_linear` directly
                 self.W_UV_O = W_UV_O.T.contiguous()
diff --git a/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe.py b/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe.py
index c9aa0ec285ba..ff381a4cc1a7 100644
--- a/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe.py
+++ b/vllm/model_executor/layers/quantization/compressed_tensors/compressed_tensors_moe.py
@@ -158,8 +158,7 @@ def process_weights_after_loading(self, layer: torch.nn.Module) -> None:
             layer.w2_input_scale = torch.nn.Parameter(
                 layer.w2_input_scale.max(), requires_grad=False)
 
-        # If rocm, normalize the weights and scales to e4m3fnuz
-        if current_platform.is_rocm():
+        if current_platform.is_fp8_fnuz():
             # Normalize the weights and scales
             w13_weight, w13_weight_scale, w13_input_scale = \
                 normalize_e4m3fn_to_e4m3fnuz(
diff --git a/vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_fp8.py b/vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_fp8.py
index aca25c9bfa19..27a74d677da7 100644
--- a/vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_fp8.py
+++ b/vllm/model_executor/layers/quantization/compressed_tensors/schemes/compressed_tensors_w8a8_fp8.py
@@ -42,7 +42,7 @@ def process_weights_after_loading(self, layer) -> None:
                 logical_widths=layer.logical_widths,
             )
 
-            if current_platform.is_rocm():
+            if current_platform.is_fp8_fnuz():
                 input_scale = getattr(layer, 'input_scale', None)
 
                 weight, max_w_scale, input_scale = normalize_e4m3fn_to_e4m3fnuz(
@@ -60,7 +60,7 @@ def process_weights_after_loading(self, layer) -> None:
         elif self.strategy == QuantizationStrategy.CHANNEL:
             weight = layer.weight
 
-            if current_platform.is_rocm():
+            if current_platform.is_fp8_fnuz():
                 input_scale = getattr(layer, 'input_scale', None)
 
                 weight, weight_scale, input_scale = \
diff --git a/vllm/model_executor/layers/quantization/fbgemm_fp8.py b/vllm/model_executor/layers/quantization/fbgemm_fp8.py
index 110e4ef2e9f1..1cc431c5cc7b 100644
--- a/vllm/model_executor/layers/quantization/fbgemm_fp8.py
+++ b/vllm/model_executor/layers/quantization/fbgemm_fp8.py
@@ -127,7 +127,7 @@ def process_weights_after_loading(self, layer: Module) -> None:
 
         weight = layer.weight
 
-        if current_platform.is_rocm():
+        if current_platform.is_fp8_fnuz():
             weight, weight_scale, input_scale = \
                 normalize_e4m3fn_to_e4m3fnuz(
                     weight=weight,
diff --git a/vllm/model_executor/layers/quantization/fp8.py b/vllm/model_executor/layers/quantization/fp8.py
index 3f8e0a2f9237..2d5d8e6adc9c 100644
--- a/vllm/model_executor/layers/quantization/fp8.py
+++ b/vllm/model_executor/layers/quantization/fp8.py
@@ -270,7 +270,7 @@ def process_weights_after_loading(self, layer: Module) -> None:
         # TODO(rob): refactor block quant into separate class.
         if self.block_quant:
             assert self.quant_config.activation_scheme == "dynamic"
-            if current_platform.is_rocm():
+            if current_platform.is_fp8_fnuz():
                 weight, weight_scale_inv, _ = \
                     normalize_e4m3fn_to_e4m3fnuz(
                         weight=layer.weight,
@@ -327,8 +327,7 @@ def process_weights_after_loading(self, layer: Module) -> None:
                 weight = layer.weight
                 weight_scale = layer.weight_scale
 
-                # If rocm, use float8_e4m3fnuz.
-                if current_platform.is_rocm():
+                if current_platform.is_fp8_fnuz():
                     weight, weight_scale, input_scale = \
                         normalize_e4m3fn_to_e4m3fnuz(
                             weight=weight,
@@ -533,7 +532,7 @@ def process_weights_after_loading(self, layer: Module) -> None:
         # TODO (rob): refactor block quant into separate class.
         if self.block_quant:
             assert self.quant_config.activation_scheme == "dynamic"
-            if current_platform.is_rocm():
+            if current_platform.is_fp8_fnuz():
                 w13_weight, w13_weight_scale_inv, w13_input_scale = \
                     normalize_e4m3fn_to_e4m3fnuz(
                         layer.w13_weight, layer.w13_weight_scale_inv,
@@ -559,9 +558,7 @@ def process_weights_after_loading(self, layer: Module) -> None:
 
         # If checkpoint is fp16, quantize in place.
         if not self.quant_config.is_checkpoint_fp8_serialized:
-            # If rocm, use float8_e4m3fnuz as dtype
-            fp8_dtype = torch.float8_e4m3fnuz \
-                        if current_platform.is_rocm() else torch.float8_e4m3fn
+            fp8_dtype = current_platform.fp8_dtype()
             w13_weight = torch.empty_like(layer.w13_weight.data,
                                           dtype=fp8_dtype)
             w2_weight = torch.empty_like(layer.w2_weight.data, dtype=fp8_dtype)
@@ -608,8 +605,7 @@ def process_weights_after_loading(self, layer: Module) -> None:
                     layer.w13_input_scale.max(), requires_grad=False)
                 layer.w2_input_scale = torch.nn.Parameter(
                     layer.w2_input_scale.max(), requires_grad=False)
-            # If rocm, normalize the weights and scales to e4m3fnuz
-            if current_platform.is_rocm():
+            if current_platform.is_fp8_fnuz():
                 # Normalize the weights and scales
                 w13_weight, w13_weight_scale, w13_input_scale = \
                     normalize_e4m3fn_to_e4m3fnuz(
diff --git a/vllm/model_executor/layers/quantization/quark/quark_moe.py b/vllm/model_executor/layers/quantization/quark/quark_moe.py
index 32dce5aaf5e0..bc26a455c6f2 100644
--- a/vllm/model_executor/layers/quantization/quark/quark_moe.py
+++ b/vllm/model_executor/layers/quantization/quark/quark_moe.py
@@ -142,8 +142,7 @@ def process_weights_after_loading(self, layer: torch.nn.Module) -> None:
             layer.w2_input_scale = torch.nn.Parameter(
                 layer.w2_input_scale.max(), requires_grad=False)
 
-        # If rocm, normalize the weights and scales to e4m3fnuz
-        if current_platform.is_rocm():
+        if current_platform.is_fp8_fnuz():
             # Normalize the weights and scales
             w13_weight, w13_weight_scale, w13_input_scale = \
                 normalize_e4m3fn_to_e4m3fnuz(
diff --git a/vllm/model_executor/layers/quantization/quark/schemes/quark_w8a8_fp8.py b/vllm/model_executor/layers/quantization/quark/schemes/quark_w8a8_fp8.py
index 7676fbddb6b8..3e4251e46931 100644
--- a/vllm/model_executor/layers/quantization/quark/schemes/quark_w8a8_fp8.py
+++ b/vllm/model_executor/layers/quantization/quark/schemes/quark_w8a8_fp8.py
@@ -39,7 +39,7 @@ def process_weights_after_loading(self, layer) -> None:
                 logical_widths=layer.logical_widths,
             )
 
-            if current_platform.is_rocm():
+            if current_platform.is_fp8_fnuz():
                 weight, max_w_scale, input_scale = normalize_e4m3fn_to_e4m3fnuz(
                     weight=weight,
                     weight_scale=max_w_scale,
@@ -55,7 +55,7 @@ def process_weights_after_loading(self, layer) -> None:
         elif self.qscheme == "per_channel":
             weight = layer.weight
 
-            if current_platform.is_rocm():
+            if current_platform.is_fp8_fnuz():
                 weight, weight_scale, input_scale = \
                     normalize_e4m3fn_to_e4m3fnuz(
                         weight=weight,
diff --git a/vllm/model_executor/layers/quantization/utils/fp8_utils.py b/vllm/model_executor/layers/quantization/utils/fp8_utils.py
index 62569185ef47..1e19302cbad8 100644
--- a/vllm/model_executor/layers/quantization/utils/fp8_utils.py
+++ b/vllm/model_executor/layers/quantization/utils/fp8_utils.py
@@ -22,10 +22,6 @@
 
 logger = init_logger(__name__)
 
-current_platform_fp8_dtype = (torch.float8_e4m3fnuz
-                              if current_platform.is_rocm() else
-                              torch.float8_e4m3fn)
-
 
 def is_fp8(x: Union[torch.dtype, torch.Tensor]) -> bool:
     if isinstance(x, torch.Tensor):
@@ -165,9 +161,7 @@ def input_to_float8(
 ) -> Tuple[torch.Tensor, torch.Tensor]:
     """This function quantizes input values to float8 values "
     "with tensor-wise quantization."""
-    if dtype is None:
-        dtype = (torch.float8_e4m3fnuz
-                 if current_platform.is_rocm() else torch.float8_e4m3fn)
+    dtype = current_platform.fp8_dtype() if dtype is None else dtype
     finfo = torch.finfo(dtype)
     min_val, max_val = x.aminmax()
     amax = torch.maximum(min_val.abs(), max_val.abs()).clamp(min=1e-12)
@@ -311,9 +305,7 @@ def per_token_group_quant_fp8(
         Tuple[torch.Tensor, torch.Tensor]: The quantized tensor and the
         scaling factor for quantization.
     """
-    if dtype is None:
-        dtype = (torch.float8_e4m3fnuz
-                 if current_platform.is_rocm() else torch.float8_e4m3fn)
+    dtype = current_platform.fp8_dtype() if dtype is None else dtype
     assert (x.shape[-1] % group_size == 0), (
         f"the last dimension of `x` {x.shape[-1]} must be divisible "
         f"by `group_size` {group_size}")
diff --git a/vllm/platforms/cuda.py b/vllm/platforms/cuda.py
index a986ec0a33ff..3897584307e9 100644
--- a/vllm/platforms/cuda.py
+++ b/vllm/platforms/cuda.py
@@ -293,6 +293,10 @@ def get_punica_wrapper(cls) -> str:
     def get_device_communicator_cls(cls) -> str:
         return "vllm.distributed.device_communicators.cuda_communicator.CudaCommunicator"  # noqa
 
+    @classmethod
+    def supports_fp8(cls) -> bool:
+        return cls.has_device_capability(89)
+
 
 # NVML utils
 # Note that NVML is not affected by `CUDA_VISIBLE_DEVICES`,
diff --git a/vllm/platforms/interface.py b/vllm/platforms/interface.py
index e7e55e11775c..7415b5d5f060 100644
--- a/vllm/platforms/interface.py
+++ b/vllm/platforms/interface.py
@@ -330,6 +330,36 @@ def get_device_communicator_cls(cls) -> str:
         """
         return "vllm.distributed.device_communicators.base_device_communicator.DeviceCommunicatorBase"  # noqa
 
+    @classmethod
+    def supports_fp8(cls) -> bool:
+        """
+        Returns whether the current platform supports FP8 types.
+        """
+        return False
+
+    @classmethod
+    def is_fp8_fnuz(cls) -> bool:
+        """
+        Returns whether the preferred FP8 type is FNUZ on the current platform.
+
+        There are two representations of FP8, OCP FP8 and FNUZ FP8.
+        The OCP specification can be found at https://tinyurl.com/b7jvwpft.
+        The FNUZ specification can be found at https://tinyurl.com/5n6hwwu5.
+
+        AMD's MI300 and MI325 have native hardware support for FNUZ. All other
+        hardware has converged on the OCP FP8 standard.
+        """
+        return False
+
+    @classmethod
+    def fp8_dtype(cls) -> torch.dtype:
+        """
+        Returns the preferred FP8 type on the current platform.
+
+        See the documentation for is_fp8_fnuz for details.
+        """
+        return torch.float8_e4m3fn
+
     @classmethod
     def use_all_gather(cls) -> bool:
         """
diff --git a/vllm/platforms/rocm.py b/vllm/platforms/rocm.py
index de4f6070f0e1..75f287b568ac 100644
--- a/vllm/platforms/rocm.py
+++ b/vllm/platforms/rocm.py
@@ -231,3 +231,20 @@ def get_current_memory_usage(cls,
     @classmethod
     def get_device_communicator_cls(cls) -> str:
         return "vllm.distributed.device_communicators.cuda_communicator.CudaCommunicator"  # noqa
+
+    @classmethod
+    def supports_fp8(cls) -> bool:
+        gcn_arch = torch.cuda.get_device_properties(0).gcnArchName
+        return any(gfx in gcn_arch for gfx in ['gfx94', 'gfx95', 'gfx12'])
+
+    @classmethod
+    def is_fp8_fnuz(cls) -> bool:
+        # only device 0 is checked, this assumes MI300 platforms are homogeneous
+        return 'gfx94' in torch.cuda.get_device_properties(0).gcnArchName
+
+    @classmethod
+    def fp8_dtype(cls) -> torch.dtype:
+        if cls.is_fp8_fnuz():
+            return torch.float8_e4m3fnuz
+        else:
+            return torch.float8_e4m3fn
diff --git a/vllm/v1/attention/backends/mla/common.py b/vllm/v1/attention/backends/mla/common.py
index 0b0f521672b0..6a7c75f43e8a 100644
--- a/vllm/v1/attention/backends/mla/common.py
+++ b/vllm/v1/attention/backends/mla/common.py
@@ -219,10 +219,11 @@
     CompressedTensorsW8A8Fp8)
 from vllm.model_executor.layers.quantization.fp8 import Fp8LinearMethod
 from vllm.model_executor.layers.quantization.utils.fp8_utils import (
-    Fp8LinearGenericOp, current_platform_fp8_dtype, is_fp8)
+    Fp8LinearGenericOp, is_fp8)
 from vllm.model_executor.layers.quantization.utils.quant_utils import (
     scaled_quantize)
 from vllm.model_executor.layers.rotary_embedding import RotaryEmbedding
+from vllm.platforms import current_platform
 from vllm.utils import cdiv, round_down
 
 try:
@@ -815,7 +816,7 @@ def get_and_maybe_dequant_weights(layer: LinearBase):
                 W_Q_UK, W_Q_UK_scales = scaled_quantize(
                     W_Q_UK,
                     self.reqaunt_weight_group_shape,
-                    quant_dtype=current_platform_fp8_dtype)
+                    quant_dtype=current_platform.fp8_dtype())
                 # For FP8 save the transpose so we can use
                 # `apply_w8a8_block_fp8_linear` directly
                 self.W_Q_UK = W_Q_UK.T.contiguous()
@@ -832,7 +833,7 @@ def get_and_maybe_dequant_weights(layer: LinearBase):
                 W_UV_O, W_UV_O_scales = scaled_quantize(
                     W_UV_O,
                     self.reqaunt_weight_group_shape,
-                    quant_dtype=current_platform_fp8_dtype)
+                    quant_dtype=current_platform.fp8_dtype())
                 # For FP8 save the transpose so we can use
                 # `apply_w8a8_block_fp8_linear` directly
                 self.W_UV_O = W_UV_O.T.contiguous()


Reviews:
user: login: ProExpertProg
body: Nice PR! Thanks for removing TODOs and cleaning up a variety of nested if statements etc.

I left a few comments for further improving the code quality - this is a pretty messy part of our codebase and we should really keep it from getting more convoluted - thanks for doing your part already!
state: CHANGES_REQUESTED
author_association: COLLABORATOR
submitted_at: 2025-03-06T20:51:22Z
user: login: jeffdaily
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T22:40:35Z
user: login: jeffdaily
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T22:50:37Z
user: login: jeffdaily
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T22:51:25Z
user: login: jeffdaily
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T22:58:44Z
user: login: jeffdaily
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T23:08:50Z
user: login: jeffdaily
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T23:28:13Z
user: login: jeffdaily
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T23:28:21Z
user: login: jeffdaily
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T23:31:58Z
user: login: jeffdaily
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T23:51:27Z
user: login: jeffdaily
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T23:52:09Z
user: login: ProExpertProg
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-06T23:54:55Z
user: login: ProExpertProg
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-06T23:57:12Z
user: login: ProExpertProg
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-06T23:58:16Z
user: login: jeffdaily
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T01:06:26Z
user: login: jeffdaily
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T01:07:30Z
user: login: jeffdaily
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T01:07:51Z
user: login: ProExpertProg
body: Thanks for addressing the comments! It looks great now.

Added couple more nits, and I think it would be nice to add a comment on the C++ side somewhere that explains there are two FP8 types on ROCm (not sure where the best place for that is). Otherwise LGTM!
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-07T01:29:44Z
user: login: tlrmchlsmth
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T01:59:54Z
user: login: jeffdaily
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T18:14:01Z
user: login: ProExpertProg
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-10T16:30:53Z
user: login: jeffdaily
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T17:09:05Z
user: login: jeffdaily
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-10T17:55:24Z
user: login: mgoin
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-10T21:44:38Z
user: login: ProExpertProg
body: I think this should be good to go!
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-11T00:46:47Z
user: login: robertgshaw2-redhat
body: This is an excellent PR
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-03-11T01:21:19Z
user: login: tlrmchlsmth
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-11T01:50:14Z
user: login: jeffdaily
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-11T19:36:33Z

Review comments:
diff_hunk: @@ -17,7 +17,7 @@
 from vllm.platforms import current_platform
 from vllm.utils import FlexibleArgumentParser
 
-FP8_DTYPE = torch.float8_e4m3fnuz if current_platform.is_rocm(
+FP8_DTYPE = torch.float8_e4m3fnuz if current_platform.is_fp8_fnuz(
 ) else torch.float8_e4m3fn
path: benchmarks/kernels/benchmark_moe.py
user: login: ProExpertProg
body: This appears in many places - could we just extract it into `current_platform.fp8_dtype()`?
created_at: 2025-03-06T20:09:10Z
author_association: COLLABORATOR
diff_hunk: @@ -14,15 +14,18 @@
 #define VLLM_DISPATCH_FLOATING_TYPES(TYPE, NAME, ...) \
   AT_DISPATCH_SWITCH(TYPE, NAME, VLLM_DISPATCH_CASE_FLOATING_TYPES(__VA_ARGS__))
 
-// TODO(luka/varun): use FP8_TYPE macro after refactoring
-#ifndef USE_ROCM
-  #define VLLM_DISPATCH_CASE_QUANT_TYPES(...)                    \
-    AT_DISPATCH_CASE(at::ScalarType::Float8_e4m3fn, __VA_ARGS__) \
-    AT_DISPATCH_CASE(at::ScalarType::Char, __VA_ARGS__)
-#else
+// ROCm devices might use either fn or fnuz, so set up dispatch table for both.
+// A host-based check at runtime will create a preferred FP8 type for ROCm
+// such that the correct kernel is dispatched.
+#ifdef USE_ROCM
   #define VLLM_DISPATCH_CASE_QUANT_TYPES(...)                      \
+    AT_DISPATCH_CASE(at::ScalarType::Float8_e4m3fn, __VA_ARGS__)   \
     AT_DISPATCH_CASE(at::ScalarType::Float8_e4m3fnuz, __VA_ARGS__) \
     AT_DISPATCH_CASE(at::ScalarType::Char, __VA_ARGS__)
+#else
path: csrc/dispatch_utils.h
user: login: ProExpertProg
body: We could add a `#define VLLM_DISPATCH_CASE_FP8_TYPES` here as well, with only one case on CUDA and 2 cases on ROCm
created_at: 2025-03-06T20:10:25Z
author_association: COLLABORATOR
diff_hunk: @@ -176,21 +176,35 @@ void rms_norm_static_fp8_quant(torch::Tensor& out,     // [..., hidden_size]
   dim3 block(std::min(hidden_size, 1024));
   const at::cuda::OptionalCUDAGuard device_guard(device_of(input));
   const cudaStream_t stream = at::cuda::getCurrentCUDAStream();
-  VLLM_DISPATCH_FLOATING_TYPES(input.scalar_type(), "rms_norm_kernel", [&] {
-    vllm::rms_norm_static_fp8_quant_kernel<scalar_t>
-        <<<grid, block, 0, stream>>>(
-            out.data_ptr<FP8_TYPE>(), input.data_ptr<scalar_t>(),
-            weight.data_ptr<scalar_t>(), scale.data_ptr<float>(), epsilon,
-            num_tokens, hidden_size);
-  });
+  if (is_fp8_ocp()) {
path: csrc/layernorm_quant_kernels.cu
user: login: ProExpertProg
body: use new dispatch macro here
created_at: 2025-03-06T20:11:15Z
author_association: COLLABORATOR
diff_hunk: @@ -227,8 +241,22 @@ void fused_add_rms_norm_static_fp8_quant(
   bool ptrs_are_aligned =
       inp_ptr % 16 == 0 && res_ptr % 16 == 0 && wt_ptr % 16 == 0;
   if (ptrs_are_aligned && hidden_size % 8 == 0) {
-    LAUNCH_FUSED_ADD_RMS_NORM(8);
+    if (is_fp8_ocp()) {
+      LAUNCH_FUSED_ADD_RMS_NORM(8, c10::Float8_e4m3fn);
+    }
+#ifdef USE_ROCM
+    else {
+      LAUNCH_FUSED_ADD_RMS_NORM(8, c10::Float8_e4m3fnuz);
+    }
+#endif
   } else {
-    LAUNCH_FUSED_ADD_RMS_NORM(0);
+    if (is_fp8_ocp()) {
+      LAUNCH_FUSED_ADD_RMS_NORM(0, c10::Float8_e4m3fn);
+    }
+#ifdef USE_ROCM
+    else {
+      LAUNCH_FUSED_ADD_RMS_NORM(0, c10::Float8_e4m3fnuz);
+    }
+#endif
path: csrc/layernorm_quant_kernels.cu
user: login: ProExpertProg
body: Could clean up dispatching logic here
created_at: 2025-03-06T20:11:51Z
author_association: COLLABORATOR
diff_hunk: @@ -7,18 +7,47 @@
 
 #ifndef USE_ROCM
   #include <c10/util/Float8_e4m3fn.h>
-using FP8_TYPE = c10::Float8_e4m3fn;
-C10_HOST_DEVICE constexpr auto FP8_E4M3_MAX =
-    std::numeric_limits<FP8_TYPE>::max();
 #else
+  #include <ATen/hip/HIPContext.h>
+  #include <c10/util/Float8_e4m3fn.h>
   #include <c10/util/Float8_e4m3fnuz.h>
   #include "amd/quant_utils.cuh"
-using FP8_TYPE = c10::Float8_e4m3fnuz;
-// Using the default max value from pytorch (240.0) will cause accuracy
-// issue when running dynamic quantization. Here use 224.0f for rocm.
-constexpr auto FP8_E4M3_MAX = 224.0f;
 #endif
-constexpr static auto kFp8Type = c10::CppTypeToScalarType<FP8_TYPE>::value;
+
+static bool is_fp8_ocp() {
+#ifndef USE_ROCM
+  return true;
+#else
+  auto dprops = at::cuda::getCurrentDeviceProperties();
+  std::string device_arch = dprops->gcnArchName;
+  size_t substring = device_arch.find("gfx94");
+  return substring == std::string::npos;
+#endif
+}
+
+template <typename T>
+class FP8_E4M3_ADJUSTED_MAX {
path: csrc/quantization/fp8/common.cuh
user: login: ProExpertProg
body: I think this should be an empty `struct`, that's more idiomatic C++, and you can avoid the explicit `public:`. Also not all upper case, that implies a constant and not class (struct) name.
created_at: 2025-03-06T20:12:58Z
author_association: COLLABORATOR
diff_hunk: @@ -7,18 +7,47 @@
 
 #ifndef USE_ROCM
   #include <c10/util/Float8_e4m3fn.h>
-using FP8_TYPE = c10::Float8_e4m3fn;
-C10_HOST_DEVICE constexpr auto FP8_E4M3_MAX =
-    std::numeric_limits<FP8_TYPE>::max();
 #else
+  #include <ATen/hip/HIPContext.h>
+  #include <c10/util/Float8_e4m3fn.h>
   #include <c10/util/Float8_e4m3fnuz.h>
   #include "amd/quant_utils.cuh"
-using FP8_TYPE = c10::Float8_e4m3fnuz;
-// Using the default max value from pytorch (240.0) will cause accuracy
-// issue when running dynamic quantization. Here use 224.0f for rocm.
-constexpr auto FP8_E4M3_MAX = 224.0f;
 #endif
-constexpr static auto kFp8Type = c10::CppTypeToScalarType<FP8_TYPE>::value;
+
+static bool is_fp8_ocp() {
+#ifndef USE_ROCM
+  return true;
+#else
+  auto dprops = at::cuda::getCurrentDeviceProperties();
+  std::string device_arch = dprops->gcnArchName;
+  size_t substring = device_arch.find("gfx94");
+  return substring == std::string::npos;
+#endif
+}
+
+template <typename T>
+class FP8_E4M3_ADJUSTED_MAX {
+ public:
+  static constexpr T val() { return {}; };
+};
+
+template <>
+class FP8_E4M3_ADJUSTED_MAX<c10::Float8_e4m3fn> {
+ public:
+  static constexpr c10::Float8_e4m3fn val() {
+    return std::numeric_limits<c10::Float8_e4m3fn>::max();
+  }
+};
+
+// Using the default max value from pytorch (240.0 0x7F) will cause accuracy
+// issue when running dynamic quantization. Here use 224.0 0x7E for rocm.
+template <>
+class FP8_E4M3_ADJUSTED_MAX<c10::Float8_e4m3fnuz> {
+ public:
+  static constexpr c10::Float8_e4m3fnuz val() {
+    return c10::Float8_e4m3fnuz(0x7E, c10::Float8_e4m3fnuz::from_bits());
+  }
+};
path: csrc/quantization/fp8/common.cuh
user: login: ProExpertProg
body: Add a "value" helper here, like
```cpp
template <typename T>
static constexpr T fp8_e4m3_adjusted_max_v = fp8_e4m3_adjusted_max<T>::value();
```
created_at: 2025-03-06T20:14:58Z
author_association: COLLABORATOR
diff_hunk: @@ -32,25 +61,35 @@ __device__ __forceinline__ float atomicMaxFloat(float* addr, float value) {
   return old;
 }
 
-template <bool is_scale_inverted>
-__device__ __forceinline__ FP8_TYPE scaled_fp8_conversion(float const val,
-                                                          float const scale) {
+template <bool is_scale_inverted, typename fp8_type>
+__device__ __forceinline__ void scaled_fp8_conversion(fp8_type& ret,
+                                                      float const val,
+                                                      float const scale) {
   float x = 0.0f;
   if constexpr (is_scale_inverted) {
     x = val * scale;
   } else {
     x = val / scale;
   }
 
-  float r = fmax(-FP8_E4M3_MAX, fmin(x, FP8_E4M3_MAX));
+  float r = fmax(-FP8_E4M3_ADJUSTED_MAX<fp8_type>::val(),
path: csrc/quantization/fp8/common.cuh
user: login: ProExpertProg
body: `fp8_e4m3_adjusted_max_v<fp8_type>`
created_at: 2025-03-06T20:15:32Z
author_association: COLLABORATOR
diff_hunk: @@ -32,25 +61,35 @@ __device__ __forceinline__ float atomicMaxFloat(float* addr, float value) {
   return old;
 }
 
-template <bool is_scale_inverted>
-__device__ __forceinline__ FP8_TYPE scaled_fp8_conversion(float const val,
-                                                          float const scale) {
+template <bool is_scale_inverted, typename fp8_type>
+__device__ __forceinline__ void scaled_fp8_conversion(fp8_type& ret,
+                                                      float const val,
+                                                      float const scale) {
   float x = 0.0f;
   if constexpr (is_scale_inverted) {
     x = val * scale;
   } else {
     x = val / scale;
   }
 
-  float r = fmax(-FP8_E4M3_MAX, fmin(x, FP8_E4M3_MAX));
+  float r = fmax(-FP8_E4M3_ADJUSTED_MAX<fp8_type>::val(),
+                 fmin(x, FP8_E4M3_ADJUSTED_MAX<fp8_type>::val()));
 #ifndef USE_ROCM
-  return static_cast<c10::Float8_e4m3fn>(r);
+  ret = static_cast<fp8_type>(r);
 #else
   // Use hardware cvt instruction for fp8 on rocm
-  return c10::Float8_e4m3fnuz(
-      __hip_cvt_float_to_fp8(r, fp8::fp8_type::__default_saturation,
-                             fp8::fp8_type::__default_interpret),
-      c10::Float8_e4m3fnuz::from_bits());
+  if constexpr (std::is_same_v<fp8_type, c10::Float8_e4m3fn>) {
+    ret =
+        fp8_type(__hip_cvt_float_to_fp8(r, __hip_fp8_e4m3::__default_saturation,
+                                        __hip_fp8_e4m3::__default_interpret),
+                 fp8_type::from_bits());
+  }
+  if constexpr (std::is_same_v<fp8_type, c10::Float8_e4m3fnuz>) {
+    ret = fp8_type(
+        __hip_cvt_float_to_fp8(r, __hip_fp8_e4m3_fnuz::__default_saturation,
+                               __hip_fp8_e4m3_fnuz::__default_interpret),
+        fp8_type::from_bits());
+  }
path: csrc/quantization/fp8/common.cuh
user: login: ProExpertProg
body: Should this be extracted to a template-specialized utility functions for possible future reuse?
created_at: 2025-03-06T20:16:36Z
author_association: COLLABORATOR
diff_hunk: @@ -32,25 +61,35 @@ __device__ __forceinline__ float atomicMaxFloat(float* addr, float value) {
   return old;
 }
 
-template <bool is_scale_inverted>
-__device__ __forceinline__ FP8_TYPE scaled_fp8_conversion(float const val,
-                                                          float const scale) {
+template <bool is_scale_inverted, typename fp8_type>
+__device__ __forceinline__ void scaled_fp8_conversion(fp8_type& ret,
path: csrc/quantization/fp8/common.cuh
user: login: ProExpertProg
body: Why the update by reference instead of returning the value?
created_at: 2025-03-06T20:17:21Z
author_association: COLLABORATOR
diff_hunk: @@ -54,16 +57,34 @@ struct ScaledQuant<
 };
 
 template <typename quant_type_t, bool is_scale_inverted>
-struct ScaledQuant<
-    quant_type_t, is_scale_inverted,
-    typename std::enable_if_t<std::is_same_v<quant_type_t, FP8_TYPE>>> {
+struct ScaledQuant<quant_type_t, is_scale_inverted,
+                   typename std::enable_if_t<
+                       std::is_same_v<quant_type_t, c10::Float8_e4m3fn>>> {
+  static __device__ __forceinline__ quant_type_t quant_fn(float const x,
+                                                          float const scale) {
+    quant_type_t ret;
+    if constexpr (is_scale_inverted) {
+      float_to_fp8(ret, x * scale);
+    } else {
+      float_to_fp8(ret, x / scale);
+    }
+    return ret;
+  }
+};
+
+template <typename quant_type_t, bool is_scale_inverted>
+struct ScaledQuant<quant_type_t, is_scale_inverted,
+                   typename std::enable_if_t<
+                       std::is_same_v<quant_type_t, c10::Float8_e4m3fnuz>>> {
path: csrc/quantization/fused_kernels/quant_conversions.cuh
user: login: ProExpertProg
body: I think theses could just stay the same struct and just expand the condition:

```suggestion
struct ScaledQuant<quant_type_t, is_scale_inverted,
                   typename std::enable_if_t<
                       std::is_same_v<quant_type_t, c10::Float8_e4m3fn> || 
                       std::is_same_v<quant_type_t, c10::Float8_e4m3fnuz>>> {
```
created_at: 2025-03-06T20:23:57Z
author_association: COLLABORATOR
diff_hunk: @@ -31,9 +31,12 @@ static __device__ __forceinline__ int8_t float_to_int8_rn(float const x) {
 #endif
 }
 
-static __device__ __forceinline__ FP8_TYPE float_to_fp8(float const x) {
-  float const r = fmax(-FP8_E4M3_MAX, fmin(x, FP8_E4M3_MAX));
-  return static_cast<FP8_TYPE>(r);
+template <typename fp8_type>
+static __device__ __forceinline__ void float_to_fp8(fp8_type& ret,
path: csrc/quantization/fused_kernels/quant_conversions.cuh
user: login: ProExpertProg
body: Same here, curious why you changed this to update by reference?
created_at: 2025-03-06T20:24:31Z
author_association: COLLABORATOR
diff_hunk: @@ -32,11 +32,11 @@ def scaled_mm_torch(a: torch.Tensor,
 
 def get_8bit_types():
     types = [torch.int8]
-    supports_fp8 = current_platform.has_device_capability(89)
-    if current_platform.is_rocm() and supports_fp8:
-        types.append(torch.float8_e4m3fnuz)
-    elif current_platform.is_cuda() and supports_fp8:
-        types.append(torch.float8_e4m3fn)
+    if current_platform.supports_fp8:
+        if current_platform.is_fp8_fnuz():
+            types.append(torch.float8_e4m3fnuz)
+        else:
+            types.append(torch.float8_e4m3fn)
path: tests/kernels/test_triton_scaled_mm.py
user: login: ProExpertProg
body: Nice simplification! I think `current_platform.fp8_type` would simplify this further
created_at: 2025-03-06T20:25:55Z
author_association: COLLABORATOR
diff_hunk: @@ -7,18 +7,47 @@
 
 #ifndef USE_ROCM
   #include <c10/util/Float8_e4m3fn.h>
-using FP8_TYPE = c10::Float8_e4m3fn;
-C10_HOST_DEVICE constexpr auto FP8_E4M3_MAX =
-    std::numeric_limits<FP8_TYPE>::max();
 #else
+  #include <ATen/hip/HIPContext.h>
+  #include <c10/util/Float8_e4m3fn.h>
   #include <c10/util/Float8_e4m3fnuz.h>
   #include "amd/quant_utils.cuh"
-using FP8_TYPE = c10::Float8_e4m3fnuz;
-// Using the default max value from pytorch (240.0) will cause accuracy
-// issue when running dynamic quantization. Here use 224.0f for rocm.
-constexpr auto FP8_E4M3_MAX = 224.0f;
 #endif
-constexpr static auto kFp8Type = c10::CppTypeToScalarType<FP8_TYPE>::value;
+
+static bool is_fp8_ocp() {
path: csrc/quantization/fp8/common.cuh
user: login: ProExpertProg
body: Could you add a brief comment about this method? I'd mention it checks device properties and .

It would also be good to check the property directly instead of querying all device properties, which is significantly slower (in my experience on CUDA, not sure if the same applies on AMD)
created_at: 2025-03-06T20:43:48Z
author_association: COLLABORATOR
diff_hunk: @@ -17,7 +17,7 @@
 from vllm.platforms import current_platform
 from vllm.utils import FlexibleArgumentParser
 
-FP8_DTYPE = torch.float8_e4m3fnuz if current_platform.is_rocm(
+FP8_DTYPE = torch.float8_e4m3fnuz if current_platform.is_fp8_fnuz(
 ) else torch.float8_e4m3fn
path: benchmarks/kernels/benchmark_moe.py
user: login: jeffdaily
body: Great idea.  But I will also keep `is_fp8_fnuz()` because it is a useful shorthand before calling `normalize_e4m3fn_to_e4m3fnuz`.
created_at: 2025-03-06T22:40:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -14,15 +14,18 @@
 #define VLLM_DISPATCH_FLOATING_TYPES(TYPE, NAME, ...) \
   AT_DISPATCH_SWITCH(TYPE, NAME, VLLM_DISPATCH_CASE_FLOATING_TYPES(__VA_ARGS__))
 
-// TODO(luka/varun): use FP8_TYPE macro after refactoring
-#ifndef USE_ROCM
-  #define VLLM_DISPATCH_CASE_QUANT_TYPES(...)                    \
-    AT_DISPATCH_CASE(at::ScalarType::Float8_e4m3fn, __VA_ARGS__) \
-    AT_DISPATCH_CASE(at::ScalarType::Char, __VA_ARGS__)
-#else
+// ROCm devices might use either fn or fnuz, so set up dispatch table for both.
+// A host-based check at runtime will create a preferred FP8 type for ROCm
+// such that the correct kernel is dispatched.
+#ifdef USE_ROCM
   #define VLLM_DISPATCH_CASE_QUANT_TYPES(...)                      \
+    AT_DISPATCH_CASE(at::ScalarType::Float8_e4m3fn, __VA_ARGS__)   \
     AT_DISPATCH_CASE(at::ScalarType::Float8_e4m3fnuz, __VA_ARGS__) \
     AT_DISPATCH_CASE(at::ScalarType::Char, __VA_ARGS__)
+#else
path: csrc/dispatch_utils.h
user: login: jeffdaily
body: I could, but where/how do you see that being used?  

We already have `VLLM_DISPATCH_CASE_QUANT_TYPES(...)` where I added the two fp8 cases on rocm while cuda still has just the one.

Then in [csrc/layernorm_quant_kernels.cu](https://github.com/vllm-project/vllm/blob/04222984f870f3e4ce92076a322e55b18427f40a/csrc/layernorm_quant_kernels.cu#L179-L185), the dispatch is based on the `input.scalar_type()` but we need to also conditionalize the dispatch based on the `out.scalar_type()`.  We're not dispatching based on the fp8 type.  

Did you mean something else?
created_at: 2025-03-06T22:50:37Z
author_association: CONTRIBUTOR
diff_hunk: @@ -14,15 +14,18 @@
 #define VLLM_DISPATCH_FLOATING_TYPES(TYPE, NAME, ...) \
   AT_DISPATCH_SWITCH(TYPE, NAME, VLLM_DISPATCH_CASE_FLOATING_TYPES(__VA_ARGS__))
 
-// TODO(luka/varun): use FP8_TYPE macro after refactoring
-#ifndef USE_ROCM
-  #define VLLM_DISPATCH_CASE_QUANT_TYPES(...)                    \
-    AT_DISPATCH_CASE(at::ScalarType::Float8_e4m3fn, __VA_ARGS__) \
-    AT_DISPATCH_CASE(at::ScalarType::Char, __VA_ARGS__)
-#else
+// ROCm devices might use either fn or fnuz, so set up dispatch table for both.
+// A host-based check at runtime will create a preferred FP8 type for ROCm
+// such that the correct kernel is dispatched.
+#ifdef USE_ROCM
   #define VLLM_DISPATCH_CASE_QUANT_TYPES(...)                      \
+    AT_DISPATCH_CASE(at::ScalarType::Float8_e4m3fn, __VA_ARGS__)   \
     AT_DISPATCH_CASE(at::ScalarType::Float8_e4m3fnuz, __VA_ARGS__) \
     AT_DISPATCH_CASE(at::ScalarType::Char, __VA_ARGS__)
+#else
path: csrc/dispatch_utils.h
user: login: jeffdaily
body: Can you nest the dispatch macros like that?
created_at: 2025-03-06T22:51:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,18 +7,47 @@
 
 #ifndef USE_ROCM
   #include <c10/util/Float8_e4m3fn.h>
-using FP8_TYPE = c10::Float8_e4m3fn;
-C10_HOST_DEVICE constexpr auto FP8_E4M3_MAX =
-    std::numeric_limits<FP8_TYPE>::max();
 #else
+  #include <ATen/hip/HIPContext.h>
+  #include <c10/util/Float8_e4m3fn.h>
   #include <c10/util/Float8_e4m3fnuz.h>
   #include "amd/quant_utils.cuh"
-using FP8_TYPE = c10::Float8_e4m3fnuz;
-// Using the default max value from pytorch (240.0) will cause accuracy
-// issue when running dynamic quantization. Here use 224.0f for rocm.
-constexpr auto FP8_E4M3_MAX = 224.0f;
 #endif
-constexpr static auto kFp8Type = c10::CppTypeToScalarType<FP8_TYPE>::value;
+
+static bool is_fp8_ocp() {
path: csrc/quantization/fp8/common.cuh
user: login: jeffdaily
body: I will add a comment, yes.

Would it ease your mind to know that `at::cuda::getCurrentDeviceProperties()` is cached internally by pytorch?

See https://github.com/pytorch/pytorch/blob/ed9c8a5d136cae1dac46c7700b83d3e07830cff8/aten/src/ATen/cuda/CUDAContext.cpp#L17-L60 for the implementation.
created_at: 2025-03-06T22:58:44Z
author_association: CONTRIBUTOR
diff_hunk: @@ -32,25 +61,35 @@ __device__ __forceinline__ float atomicMaxFloat(float* addr, float value) {
   return old;
 }
 
-template <bool is_scale_inverted>
-__device__ __forceinline__ FP8_TYPE scaled_fp8_conversion(float const val,
-                                                          float const scale) {
+template <bool is_scale_inverted, typename fp8_type>
+__device__ __forceinline__ void scaled_fp8_conversion(fp8_type& ret,
path: csrc/quantization/fp8/common.cuh
user: login: jeffdaily
body: I added `fp8_type` as a template param and I panicked that you can't have two functions that differ in just the return type in C++.



created_at: 2025-03-06T23:08:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,18 +7,47 @@
 
 #ifndef USE_ROCM
   #include <c10/util/Float8_e4m3fn.h>
-using FP8_TYPE = c10::Float8_e4m3fn;
-C10_HOST_DEVICE constexpr auto FP8_E4M3_MAX =
-    std::numeric_limits<FP8_TYPE>::max();
 #else
+  #include <ATen/hip/HIPContext.h>
+  #include <c10/util/Float8_e4m3fn.h>
   #include <c10/util/Float8_e4m3fnuz.h>
   #include "amd/quant_utils.cuh"
-using FP8_TYPE = c10::Float8_e4m3fnuz;
-// Using the default max value from pytorch (240.0) will cause accuracy
-// issue when running dynamic quantization. Here use 224.0f for rocm.
-constexpr auto FP8_E4M3_MAX = 224.0f;
 #endif
-constexpr static auto kFp8Type = c10::CppTypeToScalarType<FP8_TYPE>::value;
+
+static bool is_fp8_ocp() {
+#ifndef USE_ROCM
+  return true;
+#else
+  auto dprops = at::cuda::getCurrentDeviceProperties();
+  std::string device_arch = dprops->gcnArchName;
+  size_t substring = device_arch.find("gfx94");
+  return substring == std::string::npos;
+#endif
+}
+
+template <typename T>
+class FP8_E4M3_ADJUSTED_MAX {
path: csrc/quantization/fp8/common.cuh
user: login: jeffdaily
body: Done.
created_at: 2025-03-06T23:28:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -7,18 +7,47 @@
 
 #ifndef USE_ROCM
   #include <c10/util/Float8_e4m3fn.h>
-using FP8_TYPE = c10::Float8_e4m3fn;
-C10_HOST_DEVICE constexpr auto FP8_E4M3_MAX =
-    std::numeric_limits<FP8_TYPE>::max();
 #else
+  #include <ATen/hip/HIPContext.h>
+  #include <c10/util/Float8_e4m3fn.h>
   #include <c10/util/Float8_e4m3fnuz.h>
   #include "amd/quant_utils.cuh"
-using FP8_TYPE = c10::Float8_e4m3fnuz;
-// Using the default max value from pytorch (240.0) will cause accuracy
-// issue when running dynamic quantization. Here use 224.0f for rocm.
-constexpr auto FP8_E4M3_MAX = 224.0f;
 #endif
-constexpr static auto kFp8Type = c10::CppTypeToScalarType<FP8_TYPE>::value;
+
+static bool is_fp8_ocp() {
+#ifndef USE_ROCM
+  return true;
+#else
+  auto dprops = at::cuda::getCurrentDeviceProperties();
+  std::string device_arch = dprops->gcnArchName;
+  size_t substring = device_arch.find("gfx94");
+  return substring == std::string::npos;
+#endif
+}
+
+template <typename T>
+class FP8_E4M3_ADJUSTED_MAX {
+ public:
+  static constexpr T val() { return {}; };
+};
+
+template <>
+class FP8_E4M3_ADJUSTED_MAX<c10::Float8_e4m3fn> {
+ public:
+  static constexpr c10::Float8_e4m3fn val() {
+    return std::numeric_limits<c10::Float8_e4m3fn>::max();
+  }
+};
+
+// Using the default max value from pytorch (240.0 0x7F) will cause accuracy
+// issue when running dynamic quantization. Here use 224.0 0x7E for rocm.
+template <>
+class FP8_E4M3_ADJUSTED_MAX<c10::Float8_e4m3fnuz> {
+ public:
+  static constexpr c10::Float8_e4m3fnuz val() {
+    return c10::Float8_e4m3fnuz(0x7E, c10::Float8_e4m3fnuz::from_bits());
+  }
+};
path: csrc/quantization/fp8/common.cuh
user: login: jeffdaily
body: Done.
created_at: 2025-03-06T23:28:21Z
author_association: CONTRIBUTOR
diff_hunk: @@ -32,25 +61,35 @@ __device__ __forceinline__ float atomicMaxFloat(float* addr, float value) {
   return old;
 }
 
-template <bool is_scale_inverted>
-__device__ __forceinline__ FP8_TYPE scaled_fp8_conversion(float const val,
-                                                          float const scale) {
+template <bool is_scale_inverted, typename fp8_type>
+__device__ __forceinline__ void scaled_fp8_conversion(fp8_type& ret,
path: csrc/quantization/fp8/common.cuh
user: login: jeffdaily
body: Turns out this works just fine the way it was before and with fp8_type as a template param.  Will revert.
created_at: 2025-03-06T23:31:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -32,25 +61,35 @@ __device__ __forceinline__ float atomicMaxFloat(float* addr, float value) {
   return old;
 }
 
-template <bool is_scale_inverted>
-__device__ __forceinline__ FP8_TYPE scaled_fp8_conversion(float const val,
-                                                          float const scale) {
+template <bool is_scale_inverted, typename fp8_type>
+__device__ __forceinline__ void scaled_fp8_conversion(fp8_type& ret,
+                                                      float const val,
+                                                      float const scale) {
   float x = 0.0f;
   if constexpr (is_scale_inverted) {
     x = val * scale;
   } else {
     x = val / scale;
   }
 
-  float r = fmax(-FP8_E4M3_MAX, fmin(x, FP8_E4M3_MAX));
+  float r = fmax(-FP8_E4M3_ADJUSTED_MAX<fp8_type>::val(),
+                 fmin(x, FP8_E4M3_ADJUSTED_MAX<fp8_type>::val()));
 #ifndef USE_ROCM
-  return static_cast<c10::Float8_e4m3fn>(r);
+  ret = static_cast<fp8_type>(r);
 #else
   // Use hardware cvt instruction for fp8 on rocm
-  return c10::Float8_e4m3fnuz(
-      __hip_cvt_float_to_fp8(r, fp8::fp8_type::__default_saturation,
-                             fp8::fp8_type::__default_interpret),
-      c10::Float8_e4m3fnuz::from_bits());
+  if constexpr (std::is_same_v<fp8_type, c10::Float8_e4m3fn>) {
+    ret =
+        fp8_type(__hip_cvt_float_to_fp8(r, __hip_fp8_e4m3::__default_saturation,
+                                        __hip_fp8_e4m3::__default_interpret),
+                 fp8_type::from_bits());
+  }
+  if constexpr (std::is_same_v<fp8_type, c10::Float8_e4m3fnuz>) {
+    ret = fp8_type(
+        __hip_cvt_float_to_fp8(r, __hip_fp8_e4m3_fnuz::__default_saturation,
+                               __hip_fp8_e4m3_fnuz::__default_interpret),
+        fp8_type::from_bits());
+  }
path: csrc/quantization/fp8/common.cuh
user: login: jeffdaily
body: Done.
created_at: 2025-03-06T23:51:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -31,9 +31,12 @@ static __device__ __forceinline__ int8_t float_to_int8_rn(float const x) {
 #endif
 }
 
-static __device__ __forceinline__ FP8_TYPE float_to_fp8(float const x) {
-  float const r = fmax(-FP8_E4M3_MAX, fmin(x, FP8_E4M3_MAX));
-  return static_cast<FP8_TYPE>(r);
+template <typename fp8_type>
+static __device__ __forceinline__ void float_to_fp8(fp8_type& ret,
path: csrc/quantization/fused_kernels/quant_conversions.cuh
user: login: jeffdaily
body: Reverted.
created_at: 2025-03-06T23:52:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -17,7 +17,7 @@
 from vllm.platforms import current_platform
 from vllm.utils import FlexibleArgumentParser
 
-FP8_DTYPE = torch.float8_e4m3fnuz if current_platform.is_rocm(
+FP8_DTYPE = torch.float8_e4m3fnuz if current_platform.is_fp8_fnuz(
 ) else torch.float8_e4m3fn
path: benchmarks/kernels/benchmark_moe.py
user: login: ProExpertProg
body: Yep, agreed 😃
created_at: 2025-03-06T23:54:55Z
author_association: COLLABORATOR
diff_hunk: @@ -7,18 +7,47 @@
 
 #ifndef USE_ROCM
   #include <c10/util/Float8_e4m3fn.h>
-using FP8_TYPE = c10::Float8_e4m3fn;
-C10_HOST_DEVICE constexpr auto FP8_E4M3_MAX =
-    std::numeric_limits<FP8_TYPE>::max();
 #else
+  #include <ATen/hip/HIPContext.h>
+  #include <c10/util/Float8_e4m3fn.h>
   #include <c10/util/Float8_e4m3fnuz.h>
   #include "amd/quant_utils.cuh"
-using FP8_TYPE = c10::Float8_e4m3fnuz;
-// Using the default max value from pytorch (240.0) will cause accuracy
-// issue when running dynamic quantization. Here use 224.0f for rocm.
-constexpr auto FP8_E4M3_MAX = 224.0f;
 #endif
-constexpr static auto kFp8Type = c10::CppTypeToScalarType<FP8_TYPE>::value;
+
+static bool is_fp8_ocp() {
path: csrc/quantization/fp8/common.cuh
user: login: ProExpertProg
body: Yep, missed that this was a torch call and not a cuda/hip direct call.
created_at: 2025-03-06T23:57:12Z
author_association: COLLABORATOR
diff_hunk: @@ -14,15 +14,18 @@
 #define VLLM_DISPATCH_FLOATING_TYPES(TYPE, NAME, ...) \
   AT_DISPATCH_SWITCH(TYPE, NAME, VLLM_DISPATCH_CASE_FLOATING_TYPES(__VA_ARGS__))
 
-// TODO(luka/varun): use FP8_TYPE macro after refactoring
-#ifndef USE_ROCM
-  #define VLLM_DISPATCH_CASE_QUANT_TYPES(...)                    \
-    AT_DISPATCH_CASE(at::ScalarType::Float8_e4m3fn, __VA_ARGS__) \
-    AT_DISPATCH_CASE(at::ScalarType::Char, __VA_ARGS__)
-#else
+// ROCm devices might use either fn or fnuz, so set up dispatch table for both.
+// A host-based check at runtime will create a preferred FP8 type for ROCm
+// such that the correct kernel is dispatched.
+#ifdef USE_ROCM
   #define VLLM_DISPATCH_CASE_QUANT_TYPES(...)                      \
+    AT_DISPATCH_CASE(at::ScalarType::Float8_e4m3fn, __VA_ARGS__)   \
     AT_DISPATCH_CASE(at::ScalarType::Float8_e4m3fnuz, __VA_ARGS__) \
     AT_DISPATCH_CASE(at::ScalarType::Char, __VA_ARGS__)
+#else
path: csrc/dispatch_utils.h
user: login: ProExpertProg
body: Yes, I imagined nesting the dispatching macros. I've seen it before but we can definitely keep it this way if nesting doesn't work.
created_at: 2025-03-06T23:58:16Z
author_association: COLLABORATOR
diff_hunk: @@ -14,15 +14,18 @@
 #define VLLM_DISPATCH_FLOATING_TYPES(TYPE, NAME, ...) \
   AT_DISPATCH_SWITCH(TYPE, NAME, VLLM_DISPATCH_CASE_FLOATING_TYPES(__VA_ARGS__))
 
-// TODO(luka/varun): use FP8_TYPE macro after refactoring
-#ifndef USE_ROCM
-  #define VLLM_DISPATCH_CASE_QUANT_TYPES(...)                    \
-    AT_DISPATCH_CASE(at::ScalarType::Float8_e4m3fn, __VA_ARGS__) \
-    AT_DISPATCH_CASE(at::ScalarType::Char, __VA_ARGS__)
-#else
+// ROCm devices might use either fn or fnuz, so set up dispatch table for both.
+// A host-based check at runtime will create a preferred FP8 type for ROCm
+// such that the correct kernel is dispatched.
+#ifdef USE_ROCM
   #define VLLM_DISPATCH_CASE_QUANT_TYPES(...)                      \
+    AT_DISPATCH_CASE(at::ScalarType::Float8_e4m3fn, __VA_ARGS__)   \
     AT_DISPATCH_CASE(at::ScalarType::Float8_e4m3fnuz, __VA_ARGS__) \
     AT_DISPATCH_CASE(at::ScalarType::Char, __VA_ARGS__)
+#else
path: csrc/dispatch_utils.h
user: login: jeffdaily
body: Nesting dispatch macros was the way to go.
created_at: 2025-03-07T01:06:26Z
author_association: CONTRIBUTOR
diff_hunk: @@ -227,8 +241,22 @@ void fused_add_rms_norm_static_fp8_quant(
   bool ptrs_are_aligned =
       inp_ptr % 16 == 0 && res_ptr % 16 == 0 && wt_ptr % 16 == 0;
   if (ptrs_are_aligned && hidden_size % 8 == 0) {
-    LAUNCH_FUSED_ADD_RMS_NORM(8);
+    if (is_fp8_ocp()) {
+      LAUNCH_FUSED_ADD_RMS_NORM(8, c10::Float8_e4m3fn);
+    }
+#ifdef USE_ROCM
+    else {
+      LAUNCH_FUSED_ADD_RMS_NORM(8, c10::Float8_e4m3fnuz);
+    }
+#endif
   } else {
-    LAUNCH_FUSED_ADD_RMS_NORM(0);
+    if (is_fp8_ocp()) {
+      LAUNCH_FUSED_ADD_RMS_NORM(0, c10::Float8_e4m3fn);
+    }
+#ifdef USE_ROCM
+    else {
+      LAUNCH_FUSED_ADD_RMS_NORM(0, c10::Float8_e4m3fnuz);
+    }
+#endif
path: csrc/layernorm_quant_kernels.cu
user: login: jeffdaily
body: The nested dispatch really helped clean this up.
created_at: 2025-03-07T01:07:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -32,11 +32,11 @@ def scaled_mm_torch(a: torch.Tensor,
 
 def get_8bit_types():
     types = [torch.int8]
-    supports_fp8 = current_platform.has_device_capability(89)
-    if current_platform.is_rocm() and supports_fp8:
-        types.append(torch.float8_e4m3fnuz)
-    elif current_platform.is_cuda() and supports_fp8:
-        types.append(torch.float8_e4m3fn)
+    if current_platform.supports_fp8:
+        if current_platform.is_fp8_fnuz():
+            types.append(torch.float8_e4m3fnuz)
+        else:
+            types.append(torch.float8_e4m3fn)
path: tests/kernels/test_triton_scaled_mm.py
user: login: jeffdaily
body: Agree. Done.
created_at: 2025-03-07T01:07:51Z
author_association: CONTRIBUTOR
diff_hunk: @@ -6,6 +6,10 @@
 
 #include <torch/all.h>
 
+// need a special dispatch case macro since we will nest the FP8 dispatch
+#define AT_DISPATCH_FP8_CASE(enum_type, ...) \
+  AT_PRIVATE_CASE_TYPE_USING_HINT(enum_type, fp8_t, __VA_ARGS__)
path: csrc/dispatch_utils.h
user: login: ProExpertProg
body: This is nice!
created_at: 2025-03-07T01:14:22Z
author_association: COLLABORATOR
diff_hunk: @@ -14,17 +18,30 @@
 #define VLLM_DISPATCH_FLOATING_TYPES(TYPE, NAME, ...) \
   AT_DISPATCH_SWITCH(TYPE, NAME, VLLM_DISPATCH_CASE_FLOATING_TYPES(__VA_ARGS__))
 
-// TODO(luka/varun): use FP8_TYPE macro after refactoring
-#ifndef USE_ROCM
-  #define VLLM_DISPATCH_CASE_QUANT_TYPES(...)                    \
-    AT_DISPATCH_CASE(at::ScalarType::Float8_e4m3fn, __VA_ARGS__) \
-    AT_DISPATCH_CASE(at::ScalarType::Char, __VA_ARGS__)
-#else
+// ROCm devices might use either fn or fnuz, so set up dispatch table for both.
+// A host-based check at runtime will create a preferred FP8 type for ROCm
+// such that the correct kernel is dispatched.
+#ifdef USE_ROCM
+  #define VLLM_DISPATCH_CASE_FP8_TYPES(...)                          \
+    AT_DISPATCH_FP8_CASE(at::ScalarType::Float8_e4m3fn, __VA_ARGS__) \
+    AT_DISPATCH_FP8_CASE(at::ScalarType::Float8_e4m3fnuz, __VA_ARGS__)
+
   #define VLLM_DISPATCH_CASE_QUANT_TYPES(...)                      \
+    AT_DISPATCH_CASE(at::ScalarType::Float8_e4m3fn, __VA_ARGS__)   \
     AT_DISPATCH_CASE(at::ScalarType::Float8_e4m3fnuz, __VA_ARGS__) \
     AT_DISPATCH_CASE(at::ScalarType::Char, __VA_ARGS__)
+#else
+  #define VLLM_DISPATCH_CASE_FP8_TYPES(...) \
+    AT_DISPATCH_FP8_CASE(at::ScalarType::Float8_e4m3fn, __VA_ARGS__)
+
+  #define VLLM_DISPATCH_CASE_QUANT_TYPES(...)                    \
+    AT_DISPATCH_CASE(at::ScalarType::Float8_e4m3fn, __VA_ARGS__) \
+    AT_DISPATCH_CASE(at::ScalarType::Char, __VA_ARGS__)
 #endif
 
+#define VLLM_DISPATCH_FP8_TYPES(TYPE, NAME, ...) \
path: csrc/dispatch_utils.h
user: login: ProExpertProg
body: Maybe a quick comment above this macro that mentions that the dispatched type is available as `fp8_t` and not `scalar_t`?
created_at: 2025-03-07T01:16:22Z
author_association: COLLABORATOR
diff_hunk: @@ -7,18 +7,48 @@
 
 #ifndef USE_ROCM
   #include <c10/util/Float8_e4m3fn.h>
-using FP8_TYPE = c10::Float8_e4m3fn;
-C10_HOST_DEVICE constexpr auto FP8_E4M3_MAX =
-    std::numeric_limits<FP8_TYPE>::max();
 #else
+  #include <ATen/hip/HIPContext.h>
+  #include <c10/util/Float8_e4m3fn.h>
   #include <c10/util/Float8_e4m3fnuz.h>
   #include "amd/quant_utils.cuh"
-using FP8_TYPE = c10::Float8_e4m3fnuz;
-// Using the default max value from pytorch (240.0) will cause accuracy
-// issue when running dynamic quantization. Here use 224.0f for rocm.
-constexpr auto FP8_E4M3_MAX = 224.0f;
 #endif
-constexpr static auto kFp8Type = c10::CppTypeToScalarType<FP8_TYPE>::value;
+
+// Determines the preferred FP8 type for the current platform.
+// Note that for CUDA this just returns true,
+// but on ROCm it will check device props.
+static bool is_fp8_ocp() {
+#ifndef USE_ROCM
+  return true;
+#else
+  auto dprops = at::cuda::getCurrentDeviceProperties();
+  std::string device_arch = dprops->gcnArchName;
+  size_t substring = device_arch.find("gfx94");
+  return substring == std::string::npos;
+#endif
+}
+
+template <typename T>
+struct fp8_e4m3_adjusted_max;
+
+template <>
+struct fp8_e4m3_adjusted_max<c10::Float8_e4m3fn> {
+  static constexpr c10::Float8_e4m3fn val() {
+    return std::numeric_limits<c10::Float8_e4m3fn>::max();
+  }
+};
+
+// Using the default max value from pytorch (240.0 0x7F) will cause accuracy
+// issue when running dynamic quantization. Here use 224.0 0x7E for rocm.
path: csrc/quantization/fp8/common.cuh
user: login: ProExpertProg
body: Nit: typo (issues)
created_at: 2025-03-07T01:20:15Z
author_association: COLLABORATOR
diff_hunk: @@ -150,8 +146,7 @@ def input_to_float8(
     """This function quantizes input values to float8 values "
     "with tensor-wise quantization."""
     if dtype is None:
-        dtype = (torch.float8_e4m3fnuz
-                 if current_platform.is_rocm() else torch.float8_e4m3fn)
+        dtype = current_platform.fp8_dtype()
path: vllm/model_executor/layers/quantization/utils/fp8_utils.py
user: login: ProExpertProg
body: Nit: could be `dtype = current_platform.fp8_dtype() if dtype is None else dtype` if it fits in a line
created_at: 2025-03-07T01:23:59Z
author_association: COLLABORATOR
diff_hunk: @@ -330,6 +330,27 @@ def get_device_communicator_cls(cls) -> str:
         """
         return "vllm.distributed.device_communicators.base_device_communicator.DeviceCommunicatorBase"  # noqa
 
+    @classmethod
+    def supports_fp8(cls) -> bool:
+        """
+        Returns whether the current platform supports F8 types.
+        """
+        return False
+
+    @classmethod
+    def is_fp8_fnuz(cls) -> bool:
+        """
+        Returns whether the preferred F8 type is fnuz on the current platform.
path: vllm/platforms/interface.py
user: login: ProExpertProg
body: Maybe add a comment here about what platforms use this dtype?
created_at: 2025-03-07T01:27:55Z
author_association: COLLABORATOR
diff_hunk: @@ -330,6 +330,27 @@ def get_device_communicator_cls(cls) -> str:
         """
         return "vllm.distributed.device_communicators.base_device_communicator.DeviceCommunicatorBase"  # noqa
 
+    @classmethod
+    def supports_fp8(cls) -> bool:
path: vllm/platforms/interface.py
user: login: tlrmchlsmth
body: If we add this, we’ll need to audit other platforms — TPUs at least should support fp8 (cc @robertgshaw2-redhat @mgoin )
created_at: 2025-03-07T01:59:54Z
author_association: MEMBER
diff_hunk: @@ -330,6 +330,27 @@ def get_device_communicator_cls(cls) -> str:
         """
         return "vllm.distributed.device_communicators.base_device_communicator.DeviceCommunicatorBase"  # noqa
 
+    @classmethod
+    def supports_fp8(cls) -> bool:
+        """
+        Returns whether the current platform supports F8 types.
+        """
+        return False
+
+    @classmethod
+    def is_fp8_fnuz(cls) -> bool:
+        """
+        Returns whether the preferred F8 type is fnuz on the current platform.
path: vllm/platforms/interface.py
user: login: jeffdaily
body: Done.  I also added links to the 2 different FP8 standards for additional clarification, and the other API `fp8_dtype` refers back to the doc for `is_fp8_fnuz` to avoid copy/paste.
created_at: 2025-03-07T18:14:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -21,9 +21,7 @@
 
 logger = init_logger(__name__)
 
-current_platform_fp8_dtype = (torch.float8_e4m3fnuz
-                              if current_platform.is_rocm() else
-                              torch.float8_e4m3fn)
+current_platform_fp8_dtype = current_platform.fp8_dtype()
path: vllm/model_executor/layers/quantization/utils/fp8_utils.py
user: login: ProExpertProg
body: I think it would actually be better to get rid of this and use `current_platform.fp8_dtype()` directly
created_at: 2025-03-10T16:30:44Z
author_association: COLLABORATOR
diff_hunk: @@ -21,9 +21,7 @@
 
 logger = init_logger(__name__)
 
-current_platform_fp8_dtype = (torch.float8_e4m3fnuz
-                              if current_platform.is_rocm() else
-                              torch.float8_e4m3fn)
+current_platform_fp8_dtype = current_platform.fp8_dtype()
path: vllm/model_executor/layers/quantization/utils/fp8_utils.py
user: login: jeffdaily
body: Ok, done. Removed it again and updated corresponding code.
created_at: 2025-03-10T17:09:05Z
author_association: CONTRIBUTOR
diff_hunk: @@ -330,6 +330,27 @@ def get_device_communicator_cls(cls) -> str:
         """
         return "vllm.distributed.device_communicators.base_device_communicator.DeviceCommunicatorBase"  # noqa
 
+    @classmethod
+    def supports_fp8(cls) -> bool:
path: vllm/platforms/interface.py
user: login: jeffdaily
body: Does the audit need to happen as part of this PR or can it be a follow-up PR?

`supports_fp8` was added to replace `current_platform.has_device_capability(89)` which was incorrect on ROCm.  It was only used in tests:

- tests/quantization/test_fp8.py
- tests/kernels/test_cutlass.py
- tests/kernels/test_triton_scaled_mm.py

There should be a follow up task to audit all call sites of `has_device_capability` because those are checks for platform features and it would be better to have specific APIs for them in `current_platform`.  For example, in [gpu_worker.py](https://github.com/vllm-project/vllm/blob/bc2d4473bf943feb8abac1688dbc7c5701d7282d/vllm/v1/worker/gpu_worker.py#L276) it's used to check for bfloat16 support.  The code would improve if there was a `current_platform.supports_bfloat16()`.
created_at: 2025-03-10T17:55:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -330,6 +330,27 @@ def get_device_communicator_cls(cls) -> str:
         """
         return "vllm.distributed.device_communicators.base_device_communicator.DeviceCommunicatorBase"  # noqa
 
+    @classmethod
+    def supports_fp8(cls) -> bool:
path: vllm/platforms/interface.py
user: login: mgoin
body: TPUs do not support fp8 yet. I think we only need to consider CUDA and ROCm at the moment
created_at: 2025-03-10T21:44:38Z
author_association: MEMBER
diff_hunk: @@ -330,6 +330,27 @@ def get_device_communicator_cls(cls) -> str:
         """
         return "vllm.distributed.device_communicators.base_device_communicator.DeviceCommunicatorBase"  # noqa
 
+    @classmethod
+    def supports_fp8(cls) -> bool:
path: vllm/platforms/interface.py
user: login: ProExpertProg
body: @jeffdaily would you mind creating a GitHub issue for the device capability audit you're describing? I agree it would be a great idea and agreed it is out of the scope of this PR.
created_at: 2025-03-11T00:46:12Z
author_association: COLLABORATOR
diff_hunk: @@ -330,6 +330,27 @@ def get_device_communicator_cls(cls) -> str:
         """
         return "vllm.distributed.device_communicators.base_device_communicator.DeviceCommunicatorBase"  # noqa
 
+    @classmethod
+    def supports_fp8(cls) -> bool:
path: vllm/platforms/interface.py
user: login: jeffdaily
body: https://github.com/vllm-project/vllm/issues/14637
created_at: 2025-03-11T19:36:33Z
author_association: CONTRIBUTOR

PR comments:
user: login: github-actions[bot]
created_at: 2025-03-05T00:12:31Z
body: 👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run `fastcheck` CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your `fastcheck` build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping `simon-mo` or `khluu` to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add `ready` label to the PR or enable auto-merge.

🚀
author_association: NONE
user: login: mergify[bot]
created_at: 2025-03-05T00:12:56Z
body: This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @jeffdaily.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork
author_association: CONTRIBUTOR
user: login: jeffdaily
created_at: 2025-03-07T01:10:32Z
body: @ProExpertProg I really appreciate your thoughtful review.  It greatly improved the quality.
author_association: CONTRIBUTOR
user: login: jeffdaily
created_at: 2025-03-07T21:29:25Z
body: @ProExpertProg Getting some cuda build failures:

```
error: identifier "fp8_e4m3_adjusted_max_v< ::c10::Float8_e4m3fn> " is undefined in device code
```

Looks like fp8_e4m3_adjusted_max_v is the problem, at least for nvcc.  Any suggestions?
author_association: CONTRIBUTOR
user: login: ProExpertProg
created_at: 2025-03-07T21:36:07Z
body: Maybe add `C10_HOST_DEVICE` back in, like the original constant?
author_association: COLLABORATOR
user: login: ProExpertProg
created_at: 2025-03-10T14:01:14Z
body: @jeffdaily it looks like you missed an import in `vllm.model_executor.layers.quantization.utils.fp8_utils`: 
```
ImportError: cannot import name 'current_platform_fp8_dtype' from 'vllm.model_executor.layers.quantization.utils.fp8_utils' (/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/fp8_utils.py)
```

Could you get that fixed, and we can enable the full CI?
author_association: COLLABORATOR
user: login: jeffdaily
created_at: 2025-03-10T15:49:09Z
body: @ProExpertProg fixed.  

I had aggressively removed that variable from vllm/model_executor/layers/quantization/utils/fp8_utils.py because it wasn't used elsewhere in that file.  I failed to grep through all the other files for anyone that might have imported it.
author_association: CONTRIBUTOR
user: login: ProExpertProg
created_at: 2025-03-10T16:01:48Z
body: Yeah that was nice cleanup! @robertgshaw2-redhat @mgoin could you guys help with the fp8 support on other platforms?
author_association: COLLABORATOR
user: login: mergify[bot]
created_at: 2025-03-10T17:09:07Z
body: This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @jeffdaily.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork
author_association: CONTRIBUTOR
user: login: ProExpertProg
created_at: 2025-03-12T21:30:33Z
body: @jeffdaily I am planning to look into it more but do you know if this is a ROCm version issue? This PR is breaking the build on `main`
```
/mnt/nvme3n1p1/sage/git/nm-vllm/csrc/quantization/fp8/amd/quant_utils.cuh:25:33: error: use of undeclared identifier '__hip_fp8_e4m3'
   25 |       __hip_cvt_float_to_fp8(r, __hip_fp8_e4m3::__default_saturation,
      |                                 ^
/mnt/nvme3n1p1/sage/git/nm-vllm/csrc/quantization/fp8/amd/quant_utils.cuh:26:30: error: use of undeclared identifier '__hip_fp8_e4m3'
   26 |                              __hip_fp8_e4m3::__default_interpret),
      |                              ^
```
author_association: COLLABORATOR
user: login: jeffdaily
created_at: 2025-03-12T21:37:18Z
body: Looks like a ROCm version issue.  `__hip_fp8_e4m3` and related types were added in ROCm 6.3.  Not present in ROCm 6.2.  I was testing in a ROCm 6.3 environment and assumed vllm CI was also at the latest ROCm version.
author_association: CONTRIBUTOR
user: login: ProExpertProg
created_at: 2025-03-12T21:49:13Z
body: Could you help with the fix? I assume there's an alternative for ROCm 6.2? It's not the CI, it's a bug in a local build
author_association: COLLABORATOR
user: login: jeffdaily
created_at: 2025-03-12T22:28:43Z
body: Working on a fix now.  Can we forward-fix with a new PR or do you need to revert this one?
author_association: CONTRIBUTOR
user: login: ProExpertProg
created_at: 2025-03-12T22:41:52Z
body: I think forward fixing is fine, the revert wouldn't be immediate either anyway. Unless you think the fix will be complicated?
author_association: COLLABORATOR
user: login: jeffdaily
created_at: 2025-03-12T22:43:16Z
body: Fix shouldn't be complicated.  Will need to isolate the use of `__hip_fp8_e4m3` behind a ROCM_VERSION macro limiting it to ROCm 6.3 or newer.
author_association: CONTRIBUTOR
user: login: ProExpertProg
created_at: 2025-03-12T22:47:54Z
body: Okay ping me on vLLM Slack once done so we can merge ASAP
author_association: COLLABORATOR
user: login: jeffdaily
created_at: 2025-03-12T23:24:21Z
body: @ProExpertProg I'm not on slack yet.  Here's the PR fix.

https://github.com/vllm-project/vllm/pull/14709
author_association: CONTRIBUTOR