Repository: kokkos/kokkos
Pull request #8091: SYCL: Allow requesting out-of-order queues
Author: login: masterleinad
Created: 2025-05-16T02:26:03Z | Merged: 2025-05-20T18:44:12Z

As described in https://github.com/argonne-lcf/AuroraBugTracking/issues/22, the latest compiler drop on Aurora causes issues with in-order queues, which is what `Kokkos` uses by default.
This pull request adds a CMake option to use out-of-order queues instead. To keep changes minimal, we define `KOKKOS_IMPL_SYCL_USE_IN_ORDER_QUEUES` when ~~`KOKKOS_ENABLE_SYCL_OUT_OF_ORDER_QUEUES`~~ `KOKKOS_ENABLE_IMPL_SYCL_OUT_OF_ORDER_QUEUES` is not defined.


Diff:
diff --git a/.gitlab/hpsf-gitlab-ci.yml b/.gitlab/hpsf-gitlab-ci.yml
index f4f5ba69b76..573076ac293 100644
--- a/.gitlab/hpsf-gitlab-ci.yml
+++ b/.gitlab/hpsf-gitlab-ci.yml
@@ -144,6 +144,43 @@ INTEL-DATA-CENTER-MAX-1100:
     reports:
       junit: build/result_pvc1100.xml
 
+INTEL-DATA-CENTER-MAX-1100-OUT-OF-ORDER:
+  extends: .common-setup
+  tags: [intel-data-center-max-1100]
+  image: intel/oneapi-basekit:2024.2.1-0-devel-ubuntu22.04
+  script:
+    - apt-get update && apt-get install -y git
+    - sycl-ls
+    - export ONEAPI_DEVICE_SELECTOR=level_zero:gpu
+    - export CLANG_TIDY_EXE=$(dirname $(which icpx))/compiler/clang-tidy
+    - export CMAKE_BUILD_PARALLEL_LEVEL=48
+    - export ENV_CMAKE_OPTIONS=""
+    - export ENV_CMAKE_OPTIONS="${ENV_CMAKE_OPTIONS};-D CMAKE_BUILD_TYPE=${BUILD_TYPE}"
+    - export ENV_CMAKE_OPTIONS="${ENV_CMAKE_OPTIONS};-D CMAKE_CXX_COMPILER=icpx"
+    - export ENV_CMAKE_OPTIONS="${ENV_CMAKE_OPTIONS};-D CMAKE_EXE_LINKER_FLAGS=-fsycl -flink-huge-device-code"
+    - export ENV_CMAKE_OPTIONS="${ENV_CMAKE_OPTIONS};-D Kokkos_ENABLE_SYCL=ON"
+    - export ENV_CMAKE_OPTIONS="${ENV_CMAKE_OPTIONS};-D Kokkos_ENABLE_IMPL_SYCL_OUT_OF_ORDER_QUEUES=ON"
+    - export ENV_CMAKE_OPTIONS="${ENV_CMAKE_OPTIONS};-D Kokkos_ARCH_INTEL_PVC=ON"
+    - export ENV_CMAKE_OPTIONS="${ENV_CMAKE_OPTIONS};-D Kokkos_ENABLE_TESTS=ON"
+    - export ENV_CMAKE_OPTIONS="${ENV_CMAKE_OPTIONS};-D Kokkos_ENABLE_COMPILER_WARNINGS=ON"
+    - export ENV_CMAKE_OPTIONS="${ENV_CMAKE_OPTIONS};-D CMAKE_VERBOSE_MAKEFILE=ON"
+    - export ENV_CMAKE_OPTIONS="${ENV_CMAKE_OPTIONS};-D CMAKE_CXX_FLAGS='-Werror -fsycl-device-code-split=per_kernel -fp-model=precise'"
+    - export CTEST_BUILD_NAME="INTEL-DATA-CENTER-MAX-1100-OUT-ORDER"
+    - ctest -VV
+        -D CDASH_MODEL=${CDASH_MODEL}
+        -D GITHUB_PR_ID="${GITHUB_PR_ID}"
+        -D CMAKE_OPTIONS="${ENV_CMAKE_OPTIONS}"
+        -D OUTPUT_JUNIT_FILE=result_pvc1100_out_of_order.xml
+        -S scripts/CTestRun.cmake
+        -D CTEST_SITE="gitlab.spack.io"
+        -D CTEST_BUILD_NAME="${CTEST_BUILD_NAME}"
+  artifacts:
+    when: always
+    paths:
+      - build/result_pvc1100_out_of_order.xml
+    reports:
+      junit: build/result_pvc1100_out_of_order.xml
+
 AMD-MI300A:
   extends: .common-setup
   tags: [uo-gpu, odyssey, amd-mi300]
diff --git a/cmake/KokkosCore_config.h.in b/cmake/KokkosCore_config.h.in
index 51129b3a26a..43e6aaf3932 100644
--- a/cmake/KokkosCore_config.h.in
+++ b/cmake/KokkosCore_config.h.in
@@ -43,6 +43,7 @@
 #cmakedefine KOKKOS_ENABLE_HIP_MULTIPLE_KERNEL_INSTANTIATIONS
 #cmakedefine KOKKOS_ENABLE_IMPL_HIP_MALLOC_ASYNC
 #cmakedefine KOKKOS_ENABLE_SYCL_RELOCATABLE_DEVICE_CODE
+#cmakedefine KOKKOS_ENABLE_IMPL_SYCL_OUT_OF_ORDER_QUEUES
 #cmakedefine KOKKOS_IMPL_SYCL_DEVICE_GLOBAL_SUPPORTED
 #cmakedefine KOKKOS_ENABLE_OPENACC_FORCE_HOST_AS_DEVICE
 #cmakedefine KOKKOS_ENABLE_IMPL_HPX_ASYNC_DISPATCH
diff --git a/cmake/kokkos_enable_options.cmake b/cmake/kokkos_enable_options.cmake
index 0cc02ec81fb..cd26186c89d 100644
--- a/cmake/kokkos_enable_options.cmake
+++ b/cmake/kokkos_enable_options.cmake
@@ -54,6 +54,7 @@ endif()
 kokkos_enable_option(
   SYCL_RELOCATABLE_DEVICE_CODE ${SYCL_RDC_DEFAULT} "Whether to enable relocatable device code (RDC) for SYCL"
 )
+kokkos_enable_option(IMPL_SYCL_OUT_OF_ORDER_QUEUES OFF "Whether to make Kokkos use out-of-order queues internally")
 kokkos_enable_option(TESTS OFF "Whether to build the unit tests")
 kokkos_enable_option(BENCHMARKS OFF "Whether to build the benchmarks")
 kokkos_enable_option(EXAMPLES OFF "Whether to build the examples")
diff --git a/core/src/Kokkos_Graph.hpp b/core/src/Kokkos_Graph.hpp
index 53819d937ee..43ca685074e 100644
--- a/core/src/Kokkos_Graph.hpp
+++ b/core/src/Kokkos_Graph.hpp
@@ -193,7 +193,7 @@ decltype(auto) Graph<ExecutionSpace>::native_graph() {
   if constexpr (std::is_same_v<ExecutionSpace, Kokkos::HIP>) {
     return m_impl_ptr->hip_graph();
   }
-#elif defined(KOKKOS_ENABLE_SYCL) && defined(SYCL_EXT_ONEAPI_GRAPH)
+#elif defined(KOKKOS_ENABLE_SYCL) && defined(KOKKOS_IMPL_SYCL_GRAPH_SUPPORT)
   if constexpr (std::is_same_v<ExecutionSpace, Kokkos::SYCL>) {
     return m_impl_ptr->sycl_graph();
   }
@@ -211,7 +211,7 @@ decltype(auto) Graph<ExecutionSpace>::native_graph_exec() {
   if constexpr (std::is_same_v<ExecutionSpace, Kokkos::HIP>) {
     return m_impl_ptr->hip_graph_exec();
   }
-#elif defined(KOKKOS_ENABLE_SYCL) && defined(SYCL_EXT_ONEAPI_GRAPH)
+#elif defined(KOKKOS_ENABLE_SYCL) && defined(KOKKOS_IMPL_SYCL_GRAPH_SUPPORT)
   if constexpr (std::is_same_v<ExecutionSpace, Kokkos::SYCL>) {
     return m_impl_ptr->sycl_graph_exec();
   }
@@ -235,7 +235,7 @@ decltype(auto) Graph<ExecutionSpace>::native_graph_exec() {
 #include <HIP/Kokkos_HIP_Graph_Impl.hpp>
 #endif
 #endif
-#ifdef SYCL_EXT_ONEAPI_GRAPH
+#ifdef KOKKOS_IMPL_SYCL_GRAPH_SUPPORT
 #include <SYCL/Kokkos_SYCL_Graph_Impl.hpp>
 #endif
 #ifdef KOKKOS_IMPL_PUBLIC_INCLUDE_NOTDEFINED_GRAPH
diff --git a/core/src/Kokkos_GraphNode.hpp b/core/src/Kokkos_GraphNode.hpp
index 597559b0019..838b141d381 100644
--- a/core/src/Kokkos_GraphNode.hpp
+++ b/core/src/Kokkos_GraphNode.hpp
@@ -249,7 +249,7 @@ class GraphNodeRef {
 
 #if defined(KOKKOS_ENABLE_CUDA) ||                                           \
     (defined(KOKKOS_ENABLE_HIP) && defined(KOKKOS_IMPL_HIP_NATIVE_GRAPH)) || \
-    (defined(KOKKOS_ENABLE_SYCL) && defined(SYCL_EXT_ONEAPI_GRAPH))
+    (defined(KOKKOS_ENABLE_SYCL) && defined(KOKKOS_IMPL_SYCL_GRAPH_SUPPORT))
   template <class Functor,
             typename = std::enable_if_t<std::is_invocable_r_v<
                 void, const Kokkos::Impl::remove_cvref_t<Functor>,
@@ -260,7 +260,7 @@ class GraphNodeRef {
 #elif defined(KOKKOS_ENABLE_HIP) && defined(KOKKOS_IMPL_HIP_NATIVE_GRAPH)
   auto hip_capture(const ExecutionSpace& exec, Functor&& functor) const {
     if constexpr (std::is_same_v<ExecutionSpace, Kokkos::HIP>) {
-#elif defined(KOKKOS_ENABLE_SYCL) && defined(SYCL_EXT_ONEAPI_GRAPH)
+#elif defined(KOKKOS_ENABLE_SYCL) && defined(KOKKOS_IMPL_SYCL_GRAPH_SUPPORT)
   auto sycl_capture(const ExecutionSpace& exec, Functor&& functor) const {
     if constexpr (std::is_same_v<ExecutionSpace, Kokkos::SYCL>) {
 #endif
diff --git a/core/src/SYCL/Kokkos_SYCL.cpp b/core/src/SYCL/Kokkos_SYCL.cpp
index c7710bd6043..b703b1cafa3 100644
--- a/core/src/SYCL/Kokkos_SYCL.cpp
+++ b/core/src/SYCL/Kokkos_SYCL.cpp
@@ -114,10 +114,10 @@ void SYCL::print_configuration(std::ostream& os, bool verbose) const {
 #else
   os << "macro  KOKKOS_IMPL_SYCL_USE_IN_ORDER_QUEUES : undefined\n";
 #endif
-#ifdef SYCL_EXT_ONEAPI_GRAPH
-  os << "macro  SYCL_EXT_ONEAPI_GRAPH : defined\n";
+#ifdef KOKKOS_IMPL_SYCL_GRAPH_SUPPORT
+  os << "macro  KOKKOS_IMPL_SYCL_GRAPH_SUPPORT : defined\n";
 #else
-  os << "macro  SYCL_EXT_ONEAPI_GRAPH : undefined\n";
+  os << "macro  KOKKOS_IMPL_SYCL_GRAPH_SUPPORT : undefined\n";
 #endif
 #ifdef SYCL_EXT_ONEAPI_BFLOAT16
   os << "macro  SYCL_EXT_ONEAPI_BFLOAT16 : defined\n";
diff --git a/core/src/SYCL/Kokkos_SYCL_ParallelFor_MDRange.hpp b/core/src/SYCL/Kokkos_SYCL_ParallelFor_MDRange.hpp
index 544c4f9711b..aba4d623cb4 100644
--- a/core/src/SYCL/Kokkos_SYCL_ParallelFor_MDRange.hpp
+++ b/core/src/SYCL/Kokkos_SYCL_ParallelFor_MDRange.hpp
@@ -199,7 +199,7 @@ class Kokkos::Impl::ParallelFor<FunctorType, Kokkos::MDRangePolicy<Traits...>,
       });
     };
 
-#ifdef SYCL_EXT_ONEAPI_GRAPH
+#ifdef KOKKOS_IMPL_SYCL_GRAPH_SUPPORT
     if constexpr (Policy::is_graph_kernel::value) {
       sycl_attach_kernel_to_node(*this, cgh_lambda);
       return {};
diff --git a/core/src/SYCL/Kokkos_SYCL_ParallelFor_Range.hpp b/core/src/SYCL/Kokkos_SYCL_ParallelFor_Range.hpp
index da75f3e901d..c7eac0cd87f 100644
--- a/core/src/SYCL/Kokkos_SYCL_ParallelFor_Range.hpp
+++ b/core/src/SYCL/Kokkos_SYCL_ParallelFor_Range.hpp
@@ -138,7 +138,7 @@ class Kokkos::Impl::ParallelFor<FunctorType, Kokkos::RangePolicy<Traits...>,
       }
     };
 
-#ifdef SYCL_EXT_ONEAPI_GRAPH
+#ifdef KOKKOS_IMPL_SYCL_GRAPH_SUPPORT
     if constexpr (Policy::is_graph_kernel::value) {
       sycl_attach_kernel_to_node(*this, cgh_lambda);
       return {};
diff --git a/core/src/SYCL/Kokkos_SYCL_ParallelFor_Team.hpp b/core/src/SYCL/Kokkos_SYCL_ParallelFor_Team.hpp
index 985a23a1d95..2ecf9d2e553 100644
--- a/core/src/SYCL/Kokkos_SYCL_ParallelFor_Team.hpp
+++ b/core/src/SYCL/Kokkos_SYCL_ParallelFor_Team.hpp
@@ -113,7 +113,7 @@ class Kokkos::Impl::ParallelFor<FunctorType, Kokkos::TeamPolicy<Properties...>,
           lambda);
     };
 
-#ifdef SYCL_EXT_ONEAPI_GRAPH
+#ifdef KOKKOS_IMPL_SYCL_GRAPH_SUPPORT
     if constexpr (Policy::is_graph_kernel::value) {
       sycl_attach_kernel_to_node(*this, cgh_lambda);
       return {};
diff --git a/core/src/SYCL/Kokkos_SYCL_ParallelReduce_MDRange.hpp b/core/src/SYCL/Kokkos_SYCL_ParallelReduce_MDRange.hpp
index 6ae496caab7..e304a91ee9c 100644
--- a/core/src/SYCL/Kokkos_SYCL_ParallelReduce_MDRange.hpp
+++ b/core/src/SYCL/Kokkos_SYCL_ParallelReduce_MDRange.hpp
@@ -129,7 +129,7 @@ class Kokkos::Impl::ParallelReduce<CombinedFunctorReducerType,
         });
       };
 
-#ifdef SYCL_EXT_ONEAPI_GRAPH
+#ifdef KOKKOS_IMPL_SYCL_GRAPH_SUPPORT
       if constexpr (Policy::is_graph_kernel::value) {
         sycl_attach_kernel_to_node(*this, cgh_lambda);
       } else
@@ -306,7 +306,7 @@ class Kokkos::Impl::ParallelReduce<CombinedFunctorReducerType,
               }
             });
       };
-#ifdef SYCL_EXT_ONEAPI_GRAPH
+#ifdef KOKKOS_IMPL_SYCL_GRAPH_SUPPORT
       if constexpr (Policy::is_graph_kernel::value) {
         sycl_attach_kernel_to_node(*this, cgh_lambda);
       } else
diff --git a/core/src/SYCL/Kokkos_SYCL_ParallelReduce_Range.hpp b/core/src/SYCL/Kokkos_SYCL_ParallelReduce_Range.hpp
index 46ef974e07a..c90265de51f 100644
--- a/core/src/SYCL/Kokkos_SYCL_ParallelReduce_Range.hpp
+++ b/core/src/SYCL/Kokkos_SYCL_ParallelReduce_Range.hpp
@@ -113,7 +113,7 @@ class Kokkos::Impl::ParallelReduce<
         });
       };
 
-#ifdef SYCL_EXT_ONEAPI_GRAPH
+#ifdef KOKKOS_IMPL_SYCL_GRAPH_SUPPORT
       if constexpr (Policy::is_graph_kernel::value) {
         sycl_attach_kernel_to_node(*this, cgh_lambda);
       } else
@@ -318,7 +318,7 @@ class Kokkos::Impl::ParallelReduce<
             reduction_lambda);
       };
 
-#ifdef SYCL_EXT_ONEAPI_GRAPH
+#ifdef KOKKOS_IMPL_SYCL_GRAPH_SUPPORT
       if constexpr (Policy::is_graph_kernel::value) {
         sycl_attach_kernel_to_node(*this, cgh_lambda);
       } else
diff --git a/core/src/SYCL/Kokkos_SYCL_ParallelReduce_Team.hpp b/core/src/SYCL/Kokkos_SYCL_ParallelReduce_Team.hpp
index da9e05eb077..27c9f4147c5 100644
--- a/core/src/SYCL/Kokkos_SYCL_ParallelReduce_Team.hpp
+++ b/core/src/SYCL/Kokkos_SYCL_ParallelReduce_Team.hpp
@@ -140,7 +140,7 @@ class Kokkos::Impl::ParallelReduce<CombinedFunctorReducerType,
                 reducer.copy(device_accessible_result_ptr, &results_ptr[0]);
             });
       };
-#ifdef SYCL_EXT_ONEAPI_GRAPH
+#ifdef KOKKOS_IMPL_SYCL_GRAPH_SUPPORT
       if constexpr (Policy::is_graph_kernel::value) {
         sycl_attach_kernel_to_node(*this, cgh_lambda);
       } else
@@ -362,7 +362,7 @@ class Kokkos::Impl::ParallelReduce<CombinedFunctorReducerType,
                 sycl::range<2>(m_team_size, m_vector_size)),
             reduction_lambda);
       };
-#ifdef SYCL_EXT_ONEAPI_GRAPH
+#ifdef KOKKOS_IMPL_SYCL_GRAPH_SUPPORT
       if constexpr (Policy::is_graph_kernel::value) {
         sycl_attach_kernel_to_node(*this, cgh_lambda);
       } else
diff --git a/core/src/decl/Kokkos_Declare_SYCL.hpp b/core/src/decl/Kokkos_Declare_SYCL.hpp
index 3570ed2b6e1..2399f8a1d02 100644
--- a/core/src/decl/Kokkos_Declare_SYCL.hpp
+++ b/core/src/decl/Kokkos_Declare_SYCL.hpp
@@ -19,7 +19,7 @@
 
 #if defined(KOKKOS_ENABLE_SYCL)
 #include <SYCL/Kokkos_SYCL.hpp>
-#ifdef SYCL_EXT_ONEAPI_GRAPH
+#ifdef KOKKOS_IMPL_SYCL_GRAPH_SUPPORT
 #include <SYCL/Kokkos_SYCL_GraphNodeKernel.hpp>
 #endif
 #include <SYCL/Kokkos_SYCL_Half_Impl_Type.hpp>
diff --git a/core/src/setup/Kokkos_Setup_SYCL.hpp b/core/src/setup/Kokkos_Setup_SYCL.hpp
index 0bfdea2af1c..6e1b917287f 100644
--- a/core/src/setup/Kokkos_Setup_SYCL.hpp
+++ b/core/src/setup/Kokkos_Setup_SYCL.hpp
@@ -17,11 +17,6 @@
 #ifndef KOKKOS_SETUP_SYCL_HPP_
 #define KOKKOS_SETUP_SYCL_HPP_
 
-// FIXME_SYCL Using in-order queues currently gives better performance on Intel
-// GPUs and we run into correctness issues with out-of-order queues on NVIDIA
-// GPUs.
-#define KOKKOS_IMPL_SYCL_USE_IN_ORDER_QUEUES
-
 // FIXME_SYCL the fallback assert is temporarily disabled by default in the
 // compiler so we need to force it
 #ifndef SYCL_ENABLE_FALLBACK_ASSERT
@@ -38,6 +33,15 @@
 #include <CL/sycl.hpp>
 #endif
 
+#ifndef KOKKOS_ENABLE_IMPL_SYCL_OUT_OF_ORDER_QUEUES
+#define KOKKOS_IMPL_SYCL_USE_IN_ORDER_QUEUES
+#endif
+
+#if defined(SYCL_EXT_ONEAPI_GRAPH) && \
+    defined(KOKKOS_IMPL_SYCL_USE_IN_ORDER_QUEUES)
+#define KOKKOS_IMPL_SYCL_GRAPH_SUPPORT
+#endif
+
 // FIXME_SYCL Use type directly once it has stabilized in SYCL.
 namespace Kokkos::Impl {
 #ifndef SYCL_EXT_INTEL_USM_ADDRESS_SPACES
diff --git a/core/unit_test/CMakeLists.txt b/core/unit_test/CMakeLists.txt
index ecc9b14d5d4..22242b77363 100644
--- a/core/unit_test/CMakeLists.txt
+++ b/core/unit_test/CMakeLists.txt
@@ -861,7 +861,7 @@ if(Kokkos_ENABLE_SYCL)
     CoreUnitTest_SYCLInterOpStreamsMultiGPU SOURCES UnitTestMainInit.cpp sycl/TestSYCL_InterOp_StreamsMultiGPU.cpp
   )
 
-  if(KOKKOS_IMPL_HAVE_SYCL_EXT_ONEAPI_GRAPH)
+  if(KOKKOS_IMPL_HAVE_SYCL_EXT_ONEAPI_GRAPH AND NOT Kokkos_ENABLE_IMPL_SYCL_OUT_OF_ORDER_QUEUES)
     kokkos_add_executable_and_test(
       CoreUnitTest_SYCLInterOpGraph SOURCES UnitTestMainInit.cpp sycl/TestSYCL_InterOp_Graph.cpp
     )
diff --git a/core/unit_test/TestExecSpaceThreadSafety.hpp b/core/unit_test/TestExecSpaceThreadSafety.hpp
index 383eee8dc61..e3514678d88 100644
--- a/core/unit_test/TestExecSpaceThreadSafety.hpp
+++ b/core/unit_test/TestExecSpaceThreadSafety.hpp
@@ -95,6 +95,10 @@ TEST(TEST_CATEGORY, exec_space_thread_safety_range) {
 #ifdef KOKKOS_ENABLE_OPENMPTARGET
   if (std::is_same_v<TEST_EXECSPACE, Kokkos::Experimental::OpenMPTarget>)
     GTEST_SKIP() << "skipping since test is known to fail for OpenMPTarget";
+#endif
+#ifdef KOKKOS_ENABLE_IMPL_SYCL_OUT_OF_ORDER_QUEUES  // FIXME_SYCL
+  GTEST_SKIP()
+      << "skipping since tests are known to fail with out-of-order queues";
 #endif
   run_exec_space_thread_safety_range();
 }
@@ -138,6 +142,10 @@ TEST(TEST_CATEGORY, exec_space_thread_safety_mdrange) {
 #ifdef KOKKOS_ENABLE_OPENMPTARGET
   if (std::is_same_v<TEST_EXECSPACE, Kokkos::Experimental::OpenMPTarget>)
     GTEST_SKIP() << "skipping since test is known to fail for OpenMPTarget";
+#endif
+#ifdef KOKKOS_ENABLE_IMPL_SYCL_OUT_OF_ORDER_QUEUES  // FIXME_SYCL
+  GTEST_SKIP()
+      << "skipping since tests are known to fail with out-of-order queues";
 #endif
   run_exec_space_thread_safety_mdrange();
 }
@@ -185,6 +193,10 @@ TEST(TEST_CATEGORY, exec_space_thread_safety_team_policy) {
   if (std::is_same_v<TEST_EXECSPACE, Kokkos::Experimental::OpenMPTarget>)
     GTEST_SKIP() << "skipping for OpenMPTarget since the test is designed to "
                     "run with vector_length=1";
+#endif
+#ifdef KOKKOS_ENABLE_IMPL_SYCL_OUT_OF_ORDER_QUEUES  // FIXME_SYCL
+  GTEST_SKIP()
+      << "skipping since tests are known to fail with out-of-order queues";
 #endif
   run_exec_space_thread_safety_team_policy();
 }
@@ -224,6 +236,10 @@ TEST(TEST_CATEGORY, exec_space_thread_safety_range_reduce) {
       << "skipping OpenACC test since unsupported host-side atomics cause "
          "race conditions during shared allocation reference counting";
   THREAD_SAFETY_TEST_UNREACHABLE();
+#endif
+#ifdef KOKKOS_ENABLE_IMPL_SYCL_OUT_OF_ORDER_QUEUES  // FIXME_SYCL
+  GTEST_SKIP()
+      << "skipping since tests are known to fail with out-of-order queues";
 #endif
   run_exec_space_thread_safety_range_reduce();
 }
@@ -264,6 +280,10 @@ TEST(TEST_CATEGORY, exec_space_thread_safety_mdrange_reduce) {
       << "skipping OpenACC test since unsupported host-side atomics cause "
          "race conditions during shared allocation reference counting";
   THREAD_SAFETY_TEST_UNREACHABLE();
+#endif
+#ifdef KOKKOS_ENABLE_IMPL_SYCL_OUT_OF_ORDER_QUEUES  // FIXME_SYCL
+  GTEST_SKIP()
+      << "skipping since tests are known to fail with out-of-order queues";
 #endif
   run_exec_space_thread_safety_mdrange_reduce();
 }
@@ -316,6 +336,10 @@ TEST(TEST_CATEGORY, exec_space_thread_safety_team_policy_reduce) {
 #if defined(KOKKOS_ENABLE_SYCL) && defined(KOKKOS_IMPL_ARCH_NVIDIA_GPU)
   if (std::is_same_v<TEST_EXECSPACE, Kokkos::SYCL>)
     GTEST_SKIP() << "skipping since test is know to fail with SYCL+Cuda";
+#endif
+#ifdef KOKKOS_ENABLE_IMPL_SYCL_OUT_OF_ORDER_QUEUES  // FIXME_SYCL
+  GTEST_SKIP()
+      << "skipping since tests are known to fail with out-of-order queues";
 #endif
   run_exec_space_thread_safety_team_policy_reduce();
 }
@@ -357,6 +381,10 @@ TEST(TEST_CATEGORY, exec_space_thread_safety_range_scan) {
       << "skipping OpenACC test since unsupported host-side atomics cause "
          "race conditions during shared allocation reference counting";
   THREAD_SAFETY_TEST_UNREACHABLE();
+#endif
+#ifdef KOKKOS_ENABLE_IMPL_SYCL_OUT_OF_ORDER_QUEUES  // FIXME_SYCL
+  GTEST_SKIP()
+      << "skipping since tests are known to fail with out-of-order queues";
 #endif
   run_exec_space_thread_safety_range_scan();
 }
diff --git a/core/unit_test/TestGraph.hpp b/core/unit_test/TestGraph.hpp
index f30bbc1fd6c..0fb38ddf3b4 100644
--- a/core/unit_test/TestGraph.hpp
+++ b/core/unit_test/TestGraph.hpp
@@ -273,8 +273,8 @@ TEST_F(TEST_CATEGORY_FIXTURE(graph), submit_six) {
   if (std::is_same_v<TEST_EXECSPACE, Kokkos::Experimental::OpenMPTarget>)
     GTEST_SKIP() << "skipping since OpenMPTarget can't use team_size 1";
 #endif
-#if defined(KOKKOS_ENABLE_SYCL) &&      \
-    (!defined(SYCL_EXT_ONEAPI_GRAPH) || \
+#if defined(KOKKOS_ENABLE_SYCL) &&               \
+    (!defined(KOKKOS_IMPL_SYCL_GRAPH_SUPPORT) || \
      !defined(KOKKOS_ARCH_INTEL_GPU))  // FIXME_SYCL
   if (std::is_same_v<TEST_EXECSPACE, Kokkos::SYCL>)
     GTEST_SKIP() << "skipping since test case is known to fail with SYCL";
@@ -383,7 +383,7 @@ TEST_F(TEST_CATEGORY_FIXTURE(graph), zero_work_reduce) {
         root.then_parallel_reduce(Kokkos::RangePolicy<TEST_EXECSPACE>(0, 0),
                                   no_op_functor, count)
 #if !defined(KOKKOS_ENABLE_SYCL) || \
-    defined(SYCL_EXT_ONEAPI_GRAPH)  // FIXME_SYCL
+    defined(KOKKOS_IMPL_SYCL_GRAPH_SUPPORT)  // FIXME_SYCL
 #if !defined(KOKKOS_ENABLE_CUDA) && \
     !defined(KOKKOS_ENABLE_HIP)  // FIXME_CUDA FIXME_HIP
             .then_parallel_reduce(
@@ -816,7 +816,7 @@ struct GraphNodeTypes {
   static constexpr bool support_capture = std::is_same_v<Exec, Kokkos::Cuda>;
 #elif defined(KOKKOS_ENABLE_HIP) && defined(KOKKOS_IMPL_HIP_NATIVE_GRAPH)
     static constexpr bool support_capture = std::is_same_v<Exec, Kokkos::HIP>;
-#elif defined(KOKKOS_ENABLE_SYCL) && defined(SYCL_EXT_ONEAPI_GRAPH)
+#elif defined(KOKKOS_ENABLE_SYCL) && defined(KOKKOS_IMPL_SYCL_GRAPH_SUPPORT)
   static constexpr bool support_capture = std::is_same_v<Exec, Kokkos::SYCL>;
 #else
   static constexpr bool support_capture = false;
@@ -914,7 +914,7 @@ template <typename Exec>
 struct ExternalCapture {
 #if defined(KOKKOS_ENABLE_CUDA) ||                                           \
     (defined(KOKKOS_ENABLE_HIP) && defined(KOKKOS_IMPL_HIP_NATIVE_GRAPH)) || \
-    (defined(KOKKOS_ENABLE_SYCL) && defined(SYCL_EXT_ONEAPI_GRAPH))
+    (defined(KOKKOS_ENABLE_SYCL) && defined(KOKKOS_IMPL_SYCL_GRAPH_SUPPORT))
   // clang-format off
   template <typename Pred, typename DstType, typename... SrcTypes>
   static auto add(const Pred& pred, const Exec& exec, const DstType& dst, SrcTypes&&... srcs) {
@@ -950,7 +950,7 @@ struct ExternalCapture {
         <<<dim3(1, 1, 1), dim3(1, 1, 1), 0, exec.hip_stream()>>>(dst, srcs...);
   }
 #endif
-#if defined(KOKKOS_ENABLE_SYCL) && defined(SYCL_EXT_ONEAPI_GRAPH)
+#if defined(KOKKOS_ENABLE_SYCL) && defined(KOKKOS_IMPL_SYCL_GRAPH_SUPPORT)
   template <typename DstType, typename... SrcTypes>
   static void compute(const Kokkos::SYCL& exec, DstType* const dst,
                       const SrcTypes* const... srcs) {


Reviews:
user: login: dalg24
body: Make sure you discuss why this needs to be a public facing option and whether that is intended as a stopgap or a more long term solution.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-19T12:06:15Z
user: login: masterleinad
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T12:48:11Z
user: login: masterleinad
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T13:22:31Z
user: login: masterleinad
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T13:27:07Z
user: login: dalg24
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-19T19:01:04Z
user: login: dalg24
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-19T19:05:45Z
user: login: masterleinad
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-19T19:14:59Z
user: login: crtrott
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-19T21:42:20Z
user: login: crtrott
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-19T21:42:43Z
user: login: crtrott
body: Change the test exclusion, otherwise I am good with this (after fixing the non-changed macro name).
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-05-19T21:43:47Z
user: login: dalg24
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-20T14:08:27Z
user: login: masterleinad
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-20T14:23:40Z
user: login: dalg24
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-20T14:30:28Z
user: login: crtrott
state: APPROVED
author_association: MEMBER
submitted_at: 2025-05-20T15:01:20Z

Review comments:
diff_hunk: @@ -38,6 +33,15 @@
 #include <CL/sycl.hpp>
 #endif
 
+#ifndef KOKKOS_ENABLE_SYCL_OUT_OF_ORDER_QUEUES
+#define KOKKOS_IMPL_SYCL_USE_IN_ORDER_QUEUES
+#endif
+
+#if defined(SYCL_EXT_ONEAPI_GRAPH) && \
+    defined(KOKKOS_IMPL_SYCL_USE_IN_ORDER_QUEUES)
+#define KOKKOS_IMPL_GRAPH_SUPPORT
path: core/src/setup/Kokkos_Setup_SYCL.hpp
user: login: dalg24
body: Why no SYCL_ prefix?
Something like `KOKKOS_IMPL_SYCL_ENABLE_GRAPH`.
created_at: 2025-05-19T11:46:32Z
author_association: MEMBER
diff_hunk: @@ -17,11 +17,6 @@
 #ifndef KOKKOS_SETUP_SYCL_HPP_
 #define KOKKOS_SETUP_SYCL_HPP_
 
-// FIXME_SYCL Using in-order queues currently gives better performance on Intel
-// GPUs and we run into correctness issues with out-of-order queues on NVIDIA
-// GPUs.
path: core/src/setup/Kokkos_Setup_SYCL.hpp
user: login: dalg24
body: Does that mean the NVIDIA GPUs correctness issues are resolved?
created_at: 2025-05-19T11:49:24Z
author_association: MEMBER
diff_hunk: @@ -861,11 +861,9 @@ if(Kokkos_ENABLE_SYCL)
     CoreUnitTest_SYCLInterOpStreamsMultiGPU SOURCES UnitTestMainInit.cpp sycl/TestSYCL_InterOp_StreamsMultiGPU.cpp
   )
 
-  if(KOKKOS_IMPL_HAVE_SYCL_EXT_ONEAPI_GRAPH)
-    kokkos_add_executable_and_test(
-      CoreUnitTest_SYCLInterOpGraph SOURCES UnitTestMainInit.cpp sycl/TestSYCL_InterOp_Graph.cpp
-    )
-  endif()
+  kokkos_add_executable_and_test(
path: core/unit_test/CMakeLists.txt
user: login: dalg24
body: Why did you decide to push this into guarding the test?
We typically prefer guarding in the CMake files when possible, just like we were doing here.
created_at: 2025-05-19T11:51:26Z
author_association: MEMBER
diff_hunk: @@ -17,11 +17,6 @@
 #ifndef KOKKOS_SETUP_SYCL_HPP_
 #define KOKKOS_SETUP_SYCL_HPP_
 
-// FIXME_SYCL Using in-order queues currently gives better performance on Intel
-// GPUs and we run into correctness issues with out-of-order queues on NVIDIA
-// GPUs.
path: core/src/setup/Kokkos_Setup_SYCL.hpp
user: login: dalg24
body: Or was it these thread-safety tests that we are now skipping?
created_at: 2025-05-19T11:52:52Z
author_association: MEMBER
diff_hunk: @@ -95,6 +95,11 @@ TEST(TEST_CATEGORY, exec_space_thread_safety_range) {
 #ifdef KOKKOS_ENABLE_OPENMPTARGET
   if (std::is_same_v<TEST_EXECSPACE, Kokkos::Experimental::OpenMPTarget>)
     GTEST_SKIP() << "skipping since test is known to fail for OpenMPTarget";
+#endif
+// FIXME_SYCL
+#ifdef KOKKOS_ENABLE_SYCL_OUT_OF_ORDER_QUEUES
+  GTEST_SKIP()
+      << "skipping since tests are known to fail with out-of-order queues";
 #endif
path: core/unit_test/TestExecSpaceThreadSafety.hpp
user: login: dalg24
body: Does that mean that thread safety is currently compromised with OOO queues?
created_at: 2025-05-19T11:53:43Z
author_association: MEMBER
diff_hunk: @@ -95,6 +95,11 @@ TEST(TEST_CATEGORY, exec_space_thread_safety_range) {
 #ifdef KOKKOS_ENABLE_OPENMPTARGET
   if (std::is_same_v<TEST_EXECSPACE, Kokkos::Experimental::OpenMPTarget>)
     GTEST_SKIP() << "skipping since test is known to fail for OpenMPTarget";
+#endif
+// FIXME_SYCL
+#ifdef KOKKOS_ENABLE_SYCL_OUT_OF_ORDER_QUEUES
path: core/unit_test/TestExecSpaceThreadSafety.hpp
user: login: dalg24
body: (Optional)
```suggestion
#ifdef KOKKOS_ENABLE_SYCL_OUT_OF_ORDER_QUEUES  // FIXME_SYCL
```
Same comment everywhere.
It comes handy if you grep FIXME_SYCL in the code base.
You get more information without having to jump to the matches.
created_at: 2025-05-19T11:54:16Z
author_association: MEMBER
diff_hunk: @@ -54,6 +54,7 @@ endif()
 kokkos_enable_option(
   SYCL_RELOCATABLE_DEVICE_CODE ${SYCL_RDC_DEFAULT} "Whether to enable relocatable device code (RDC) for SYCL"
 )
+kokkos_enable_option(SYCL_OUT_OF_ORDER_QUEUES OFF "Whether to make Kokkos use out-of-order queues internally")
path: cmake/kokkos_enable_options.cmake
user: login: dalg24
body: Are they issues not tied to toolchains versions?
Please explain why we want/need to push that choice onto the user.
created_at: 2025-05-19T12:04:43Z
author_association: MEMBER
diff_hunk: @@ -95,6 +95,11 @@ TEST(TEST_CATEGORY, exec_space_thread_safety_range) {
 #ifdef KOKKOS_ENABLE_OPENMPTARGET
   if (std::is_same_v<TEST_EXECSPACE, Kokkos::Experimental::OpenMPTarget>)
     GTEST_SKIP() << "skipping since test is known to fail for OpenMPTarget";
+#endif
+// FIXME_SYCL
+#ifdef KOKKOS_ENABLE_SYCL_OUT_OF_ORDER_QUEUES
+  GTEST_SKIP()
+      << "skipping since tests are known to fail with out-of-order queues";
 #endif
path: core/unit_test/TestExecSpaceThreadSafety.hpp
user: login: masterleinad
body: Yes. I haven't investigated further.
created_at: 2025-05-19T12:48:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -17,11 +17,6 @@
 #ifndef KOKKOS_SETUP_SYCL_HPP_
 #define KOKKOS_SETUP_SYCL_HPP_
 
-// FIXME_SYCL Using in-order queues currently gives better performance on Intel
-// GPUs and we run into correctness issues with out-of-order queues on NVIDIA
-// GPUs.
path: core/src/setup/Kokkos_Setup_SYCL.hpp
user: login: masterleinad
body: Haven't investigated further, only tested on Intel GPUs again.
created_at: 2025-05-19T13:22:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -861,11 +861,9 @@ if(Kokkos_ENABLE_SYCL)
     CoreUnitTest_SYCLInterOpStreamsMultiGPU SOURCES UnitTestMainInit.cpp sycl/TestSYCL_InterOp_StreamsMultiGPU.cpp
   )
 
-  if(KOKKOS_IMPL_HAVE_SYCL_EXT_ONEAPI_GRAPH)
-    kokkos_add_executable_and_test(
-      CoreUnitTest_SYCLInterOpGraph SOURCES UnitTestMainInit.cpp sycl/TestSYCL_InterOp_Graph.cpp
-    )
-  endif()
+  kokkos_add_executable_and_test(
path: core/unit_test/CMakeLists.txt
user: login: masterleinad
body: I found it cleaner to later decide in a preprocessor define when we actually want to allow Kokkos to use SYCL native implementation instead of duplicating the logic here. The alternative is to define a CMake variable based on the availability of the implementation and whether in-order queues are used and use that with `cmakedefine`.
created_at: 2025-05-19T13:27:07Z
author_association: CONTRIBUTOR
path: cmake/kokkos_tribits.cmake
user: login: dalg24
body: Did you mean to commit this?
created_at: 2025-05-19T19:01:04Z
author_association: MEMBER
diff_hunk: @@ -861,11 +861,9 @@ if(Kokkos_ENABLE_SYCL)
     CoreUnitTest_SYCLInterOpStreamsMultiGPU SOURCES UnitTestMainInit.cpp sycl/TestSYCL_InterOp_StreamsMultiGPU.cpp
   )
 
-  if(KOKKOS_IMPL_HAVE_SYCL_EXT_ONEAPI_GRAPH)
-    kokkos_add_executable_and_test(
-      CoreUnitTest_SYCLInterOpGraph SOURCES UnitTestMainInit.cpp sycl/TestSYCL_InterOp_Graph.cpp
-    )
-  endif()
+  kokkos_add_executable_and_test(
path: core/unit_test/CMakeLists.txt
user: login: dalg24
body: I dislike listing test files that compiles to nothing.
I would have rather guarded for `if(KOKKOS_IMPL_HAVE_SYCL_EXT_ONEAPI_GRAPH AND KOKKOS_ENABLE_IMPL_SYCL_OUT_OF_ORDER_QUEUES)`.
I am not blocking on this.  We can see how others feel about this.
created_at: 2025-05-19T19:05:45Z
author_association: MEMBER
path: cmake/kokkos_tribits.cmake
user: login: masterleinad
body: No, fixed.
created_at: 2025-05-19T19:14:58Z
author_association: CONTRIBUTOR
diff_hunk: @@ -38,6 +33,15 @@
 #include <CL/sycl.hpp>
 #endif
 
+#ifndef KOKKOS_ENABLE_IMPL_SYCL_OUT_OF_ORDER_QUEUES
+#define KOKKOS_IMPL_SYCL_USE_IN_ORDER_QUEUES
+#endif
+
+#if defined(SYCL_EXT_ONEAPI_GRAPH) && \
+    defined(KOKKOS_IMPL_SYCL_USE_IN_ORDER_QUEUES)
+#define KOKKOS_IMPL_GRAPH_SUPPORT
path: core/src/setup/Kokkos_Setup_SYCL.hpp
user: login: crtrott
body: ```suggestion
#define KOKKOS_IMPL_SYCL_GRAPH_SUPPORT
```
Don't you use `KOKKOS_IMPL_SYCL_GRAPH_SUPPORT` everywhere else?
created_at: 2025-05-19T21:42:20Z
author_association: MEMBER
diff_hunk: @@ -861,11 +861,9 @@ if(Kokkos_ENABLE_SYCL)
     CoreUnitTest_SYCLInterOpStreamsMultiGPU SOURCES UnitTestMainInit.cpp sycl/TestSYCL_InterOp_StreamsMultiGPU.cpp
   )
 
-  if(KOKKOS_IMPL_HAVE_SYCL_EXT_ONEAPI_GRAPH)
-    kokkos_add_executable_and_test(
-      CoreUnitTest_SYCLInterOpGraph SOURCES UnitTestMainInit.cpp sycl/TestSYCL_InterOp_Graph.cpp
-    )
-  endif()
+  kokkos_add_executable_and_test(
path: core/unit_test/CMakeLists.txt
user: login: crtrott
body: Agreed
created_at: 2025-05-19T21:42:43Z
author_association: MEMBER
diff_hunk: @@ -114,10 +114,10 @@ void SYCL::print_configuration(std::ostream& os, bool verbose) const {
 #else
   os << "macro  KOKKOS_IMPL_SYCL_USE_IN_ORDER_QUEUES : undefined\n";
 #endif
-#ifdef SYCL_EXT_ONEAPI_GRAPH
path: core/src/SYCL/Kokkos_SYCL.cpp
user: login: dalg24
body: Don't you care to see whether that macro is defined regardless, as information about the underlying toolchain?
I do realize you could in principle deduce its values from the configuration that is printed, I am asking if you don't deem it useful to have it directly.
created_at: 2025-05-20T14:02:56Z
author_association: MEMBER
diff_hunk: @@ -114,10 +114,10 @@ void SYCL::print_configuration(std::ostream& os, bool verbose) const {
 #else
   os << "macro  KOKKOS_IMPL_SYCL_USE_IN_ORDER_QUEUES : undefined\n";
 #endif
-#ifdef SYCL_EXT_ONEAPI_GRAPH
path: core/src/SYCL/Kokkos_SYCL.cpp
user: login: masterleinad
body: I think `KOKKOS_IMPL_SYCL_GRAPH_SUPPORT` is more useful enough to report on its own. `SYCL_EXT_ONEAPI_GRAPH` isn't used on its own anymore in `Kokkos`.
created_at: 2025-05-20T14:23:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -114,10 +114,10 @@ void SYCL::print_configuration(std::ostream& os, bool verbose) const {
 #else
   os << "macro  KOKKOS_IMPL_SYCL_USE_IN_ORDER_QUEUES : undefined\n";
 #endif
-#ifdef SYCL_EXT_ONEAPI_GRAPH
path: core/src/SYCL/Kokkos_SYCL.cpp
user: login: dalg24
body: > `SYCL_EXT_ONEAPI_GRAPH` isn't used on its own anymore in `Kokkos`.

Well you do use it in `Kokkos_Setup_SYCL.hpp` when you decide whether to define `KOKKOS_IMPL_SYCL_GRAPH_SUPPORT`.

Anyway I am not blocking on this.
created_at: 2025-05-20T14:30:22Z
author_association: MEMBER

PR comments:
user: login: masterleinad
created_at: 2025-05-19T12:41:35Z
body: > Make sure you discuss why this needs to be a public facing option and whether that is intended as a stopgap or a more long term solution.

We were previously seeing better performance with in-order queues due to fewer events being created. It's important for downstream apps to know if the default queue used by `Kokkos` is in-order or out-of-order. Adding an option makes that clearer. Still using in-order queues is recommended at this point and thus stays the default. Adding a build also makes sure that we actually test code that allows out-of-order queues.
On the other hand, this pull request clearly is somewhat of a stop gap solution. It's not clear if it can be mitigated by setting other environment variables and what the best workaround is.
author_association: CONTRIBUTOR
user: login: dalg24
created_at: 2025-05-20T14:34:37Z
body: Make sure you update the description
author_association: MEMBER