Repository: intel/llvm
Pull request #18197: [SYCL][E2E] Enable llvm-lit to accept device architecture
Author: login: DBDuncan
Created: 2025-04-25T13:51:45Z | Merged: 2025-05-09T15:57:20Z

Allow for selection of specific devices via `level_zero:arch-intel_gpu_bmg_g21` which accepts device architectures listed under `sycl-ls --verbose`. Devices with multiple GPU's under the devices with multiple GPU's under the device selection style of `backend:*device type*` such as `level_zero:gpu` can cause incorrect aspects to be marked as unsupported in the tests ran through `llvm-lit` using `// REQUIRES: aspect-`. In addition, only the first device listed via `sycl-ls` under the label is ran.
    
This occurs because using the device selection style of `level_zero:gpu` there is currently no way to choose a specific device that falls under that category. Such as a battlemage GPU and an intel iGPU. `sycl/test-e2e/lit.cfg.py` takes all available device aspects and marks aspects as supported that are available for all the devices. This is problematic for aspects such as `ext_oneapi_bindless_images` which is supported on battlemage but not Intel iGPU's. Even if battlemage, which supports bindless images is used to run all the tests as it comes first in the device list shown under `level_zero:gpu`, tests requiring that aspect will not run.


Diff:
diff --git a/sycl/test-e2e/README.md b/sycl/test-e2e/README.md
index ca48be8a961c6..12f52decb456d 100644
--- a/sycl/test-e2e/README.md
+++ b/sycl/test-e2e/README.md
@@ -177,6 +177,13 @@ separated from comma-separated list of target devices with colon. Example:
 -DSYCL_TEST_E2E_TARGETS="opencl:cpu;level_zero:gpu;cuda:gpu;hip:gpu"
 ```
 
+In addition, device architecture as shown in sycl-ls is accepted with the
+"arch-" prefix. Example:
+
+```bash
+-DSYCL_TEST_E2E_TARGETS="cuda:arch-nvidia_gpu_sm_61;level_zero:arch-intel_gpu_bmg_b21"
+```
+
 ***OpenCL_LIBRARY*** - path to OpenCL ICD loader library. OpenCL
 interoperability tests require OpenCL ICD loader to be linked with. For such
 tests OpenCL ICD loader library should be installed in the system or available
diff --git a/sycl/test-e2e/lit.cfg.py b/sycl/test-e2e/lit.cfg.py
index 59f8e890fed8e..be45ef7c2a3de 100644
--- a/sycl/test-e2e/lit.cfg.py
+++ b/sycl/test-e2e/lit.cfg.py
@@ -191,6 +191,7 @@
 # Disable the UR logger callback sink during test runs as output to SYCL RT can interfere with some tests relying on standard input/output
 llvm_config.with_environment("UR_LOG_CALLBACK", "disabled")
 
+
 # Temporarily modify environment to be the same that we use when running tests
 class test_env:
     def __enter__(self):
@@ -276,6 +277,7 @@ def quote_path(path):
         return f'"{path}"'
     return shlex.quote(path)
 
+
 # Call the function to perform the check and add the feature
 check_igc_tag_and_add_feature()
 
@@ -295,6 +297,7 @@ def quote_path(path):
 if lit_config.params.get("spirv-backend", False):
     config.available_features.add("spirv-backend")
 
+
 # Use this to make sure that any dynamic checks below are done in the build
 # directory and not where the sources are located. This is important for the
 # in-tree configuration (as opposite to the standalone one).
@@ -634,22 +637,25 @@ def open_check_file(file_name):
     if "amdgcn" in sp[1]:
         config.sycl_build_targets.add("target-amd")
 
-cmd = "{} {}".format(config.run_launcher, sycl_ls) if config.run_launcher else sycl_ls
-sycl_ls_output = subprocess.check_output(cmd, text=True, shell=True)
-
-# In contrast to `cpu` feature this is a compile-time feature, which is needed
-# to check if we can build cpu AOT tests.
-if "opencl:cpu" in sycl_ls_output:
-    config.available_features.add("opencl-cpu-rt")
-
-if len(config.sycl_devices) == 1 and config.sycl_devices[0] == "all":
-    devices = set()
-    for line in sycl_ls_output.splitlines():
-        if not line.startswith("["):
-            continue
-        (backend, device) = line[1:].split("]")[0].split(":")
-        devices.add("{}:{}".format(backend, device))
-    config.sycl_devices = list(devices)
+with test_env():
+    cmd = (
+        "{} {}".format(config.run_launcher, sycl_ls) if config.run_launcher else sycl_ls
+    )
+    sycl_ls_output = subprocess.check_output(cmd, text=True, shell=True)
+
+    # In contrast to `cpu` feature this is a compile-time feature, which is needed
+    # to check if we can build cpu AOT tests.
+    if "opencl:cpu" in sycl_ls_output:
+        config.available_features.add("opencl-cpu-rt")
+
+    if len(config.sycl_devices) == 1 and config.sycl_devices[0] == "all":
+        devices = set()
+        for line in sycl_ls_output.splitlines():
+            if not line.startswith("["):
+                continue
+            (backend, device) = line[1:].split("]")[0].split(":")
+            devices.add("{}:{}".format(backend, device))
+        config.sycl_devices = list(devices)
 
 if len(config.sycl_devices) > 1:
     lit_config.note(
@@ -670,7 +676,11 @@ def remove_level_zero_suffix(devices):
 }
 for d in remove_level_zero_suffix(config.sycl_devices):
     be, dev = d.split(":")
-    if be not in available_devices or dev not in available_devices[be]:
+    # Verify platform
+    if be not in available_devices:
+        lit_config.error("Unsupported device {}".format(d))
+    # Verify device from available_devices or accept if contains "arch-"
+    if dev not in available_devices[be] and not "arch-" in dev:
         lit_config.error("Unsupported device {}".format(d))
 
 if "cuda:gpu" in config.sycl_devices:
@@ -828,6 +838,79 @@ def remove_level_zero_suffix(devices):
 if config.test_mode != "build-only":
     config.sycl_build_targets = set()
 
+
+def get_sycl_ls_verbose(sycl_device, env):
+    with test_env():
+        # When using the ONEAPI_DEVICE_SELECTOR environment variable, sycl-ls
+        # prints warnings that might derail a user thinking something is wrong
+        # with their test run. It's just us filtering here, so silence them unless
+        # we get an exit status.
+        try:
+            cmd = "{} {} --verbose".format(config.run_launcher or "", sycl_ls)
+            sp = subprocess.run(
+                cmd, env=env, text=True, shell=True, capture_output=True
+            )
+            sp.check_returncode()
+        except subprocess.CalledProcessError as e:
+            # capturing e allows us to see path resolution errors / system
+            # permissions errors etc
+            lit_config.fatal(
+                f"Cannot find devices under {sycl_device}\n"
+                f"{e}\n"
+                f"stdout:{sp.stdout}\n"
+                f"stderr:{sp.stderr}\n"
+            )
+        return sp.stdout.splitlines()
+
+
+# A device filter such as level_zero:gpu can have multiple devices under it and
+# the order is not guaranteed. The aspects enabled are also restricted to what
+# is supported on all devices under the label. It is possible for level_zero:gpu
+# and level_zero:0 to select different devices on different machines with the
+# same hardware. It is not currently possible to pass the device architecture to
+# ONEAPI_DEVICE_SELECTOR. Instead, if "BACKEND:arch-DEVICE_ARCH" is provided to
+# "sycl_devices", giving the desired device architecture, select a device that
+# matches that architecture using the backend:device-num device selection
+# scheme.
+filtered_sycl_devices = []
+for sycl_device in remove_level_zero_suffix(config.sycl_devices):
+    backend, device_arch = sycl_device.split(":", 1)
+
+    if not "arch-" in device_arch:
+        filtered_sycl_devices.append(sycl_device)
+        continue
+
+    env = copy.copy(llvm_config.config.environment)
+
+    # Find all available devices under the backend
+    env["ONEAPI_DEVICE_SELECTOR"] = backend + ":*"
+
+    detected_architectures = []
+
+    for line in get_sycl_ls_verbose(backend + ":*", env):
+        if re.match(r" *Architecture:", line):
+            _, architecture = line.strip().split(":", 1)
+            detected_architectures.append(architecture.strip())
+
+    device = device_arch.replace("arch-", "")
+
+    if device in detected_architectures:
+        device_num = detected_architectures.index(device)
+        filtered_sycl_devices.append(backend + ":" + str(device_num))
+    else:
+        lit_config.warning(
+            "Couldn't find device with architecture {}"
+            " under {} device selector! Skipping device "
+            "{}".format(device, backend + ":*", sycl_device)
+        )
+
+if not filtered_sycl_devices and not config.test_mode == "build-only":
+    lit_config.error(
+        "No sycl devices selected! Check your device " "architecture filters."
+    )
+
+config.sycl_devices = filtered_sycl_devices
+
 for sycl_device in remove_level_zero_suffix(config.sycl_devices):
     be, dev = sycl_device.split(":")
     config.available_features.add("any-device-is-" + dev)
@@ -856,23 +939,6 @@ def remove_level_zero_suffix(devices):
     env["ONEAPI_DEVICE_SELECTOR"] = sycl_device
     if sycl_device.startswith("cuda:"):
         env["SYCL_UR_CUDA_ENABLE_IMAGE_SUPPORT"] = "1"
-    # When using the ONEAPI_DEVICE_SELECTOR environment variable, sycl-ls
-    # prints warnings that might derail a user thinking something is wrong
-    # with their test run. It's just us filtering here, so silence them unless
-    # we get an exit status.
-    try:
-        cmd = "{} {} --verbose".format(config.run_launcher or "", sycl_ls)
-        sp = subprocess.run(cmd, env=env, text=True, shell=True, capture_output=True)
-        sp.check_returncode()
-    except subprocess.CalledProcessError as e:
-        # capturing e allows us to see path resolution errors / system
-        # permissions errors etc
-        lit_config.fatal(
-            f"Cannot list device aspects for {sycl_device}\n"
-            f"{e}\n"
-            f"stdout:{sp.stdout}\n"
-            f"stderr:{sp.stderr}\n"
-        )
 
     dev_aspects = []
     dev_sg_sizes = []
@@ -880,7 +946,7 @@ def remove_level_zero_suffix(devices):
     # See format.py's parse_min_intel_driver_req for explanation.
     is_intel_driver = False
     intel_driver_ver = {}
-    for line in sp.stdout.splitlines():
+    for line in get_sycl_ls_verbose(sycl_device, env):
         if re.match(r" *Vendor *: Intel\(R\) Corporation", line):
             is_intel_driver = True
         if re.match(r" *Driver *:", line):


Reviews:
user: login: aelovikov-intel
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-08T21:11:50Z
user: login: DBDuncan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T14:56:09Z
user: login: DBDuncan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T14:56:17Z
user: login: DBDuncan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T14:56:36Z
user: login: aelovikov-intel
body: Good enough to me, none of us are python experts (to the best of my knowledge).

@sarnex ?
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T15:00:34Z
user: login: sarnex
body: also not a python export but lgtm minus a minor question
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T15:24:24Z
user: login: aelovikov-intel
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T15:27:33Z
user: login: sarnex
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T15:33:36Z
user: login: DBDuncan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T15:36:11Z
user: login: DBDuncan
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T15:37:49Z
user: login: sarnex
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-05-09T15:42:46Z

Review comments:
diff_hunk: @@ -821,6 +826,70 @@ def remove_level_zero_suffix(devices):
 if config.test_mode != "build-only":
     config.sycl_build_targets = set()
 
+# A device filter such as level_zero:gpu can have multiple devices under it and
+# the order is not guaranteed. The aspects enabled are also restricted to what
+# is supported on all devices under the label. It is possible for level_zero:gpu
+# and level_zero:0 to select different devices on different machines with the
+# same hardware. It is not currently possible to pass the device architecture to
+# ONEAPI_DEVICE_SELECTOR. Instead, if "BACKEND:arch-DEVICE_ARCH" is provided to
+# "sycl_devices", giving the desired device architecture, select a device that
+# matches that architecture using the backend:device-num device selection
+# scheme.
+filtered_sycl_devices = []
+for sycl_device in remove_level_zero_suffix(config.sycl_devices):
+
+    backend, device_arch  = sycl_device.split(":", 1)
+
+    if ("arch-" in device_arch):
path: sycl/test-e2e/lit.cfg.py
user: login: aelovikov-intel
body: Invert the condition and do early continue. Alternatively, outline arch-processing and use lists comprehension like
```
tmp = [process_arch(sycl_dev) if sycl_dev.contains(":arch-") else sycl_dev
       for sycl_dev in remove_level_zero_suffix(config.sycl_devices)]
```
created_at: 2025-05-08T21:02:14Z
author_association: CONTRIBUTOR
diff_hunk: @@ -821,6 +826,70 @@ def remove_level_zero_suffix(devices):
 if config.test_mode != "build-only":
     config.sycl_build_targets = set()
 
+# A device filter such as level_zero:gpu can have multiple devices under it and
+# the order is not guaranteed. The aspects enabled are also restricted to what
+# is supported on all devices under the label. It is possible for level_zero:gpu
+# and level_zero:0 to select different devices on different machines with the
+# same hardware. It is not currently possible to pass the device architecture to
+# ONEAPI_DEVICE_SELECTOR. Instead, if "BACKEND:arch-DEVICE_ARCH" is provided to
+# "sycl_devices", giving the desired device architecture, select a device that
+# matches that architecture using the backend:device-num device selection
+# scheme.
+filtered_sycl_devices = []
+for sycl_device in remove_level_zero_suffix(config.sycl_devices):
+
+    backend, device_arch  = sycl_device.split(":", 1)
+
+    if ("arch-" in device_arch):
+        env = copy.copy(llvm_config.config.environment)
+
+        # Find all available devices under the backend
+        env["ONEAPI_DEVICE_SELECTOR"] = backend + ":*"
+
+        # When using the ONEAPI_DEVICE_SELECTOR environment variable, sycl-ls
+        # prints warnings that might derail a user thinking something is wrong
+        # with their test run. It's just us filtering here, so silence them unless
+        # we get an exit status.
path: sycl/test-e2e/lit.cfg.py
user: login: aelovikov-intel
body: Please outline to some helper instead of copy-pasting. While doing this please take a look `with test_env():`, it might be beneficial/necessary here and in the pre-existing `sycl-ls` invocation.
created_at: 2025-05-08T21:04:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -821,6 +826,70 @@ def remove_level_zero_suffix(devices):
 if config.test_mode != "build-only":
     config.sycl_build_targets = set()
 
+# A device filter such as level_zero:gpu can have multiple devices under it and
+# the order is not guaranteed. The aspects enabled are also restricted to what
+# is supported on all devices under the label. It is possible for level_zero:gpu
+# and level_zero:0 to select different devices on different machines with the
+# same hardware. It is not currently possible to pass the device architecture to
+# ONEAPI_DEVICE_SELECTOR. Instead, if "BACKEND:arch-DEVICE_ARCH" is provided to
+# "sycl_devices", giving the desired device architecture, select a device that
+# matches that architecture using the backend:device-num device selection
+# scheme.
+filtered_sycl_devices = []
+for sycl_device in remove_level_zero_suffix(config.sycl_devices):
+
+    backend, device_arch  = sycl_device.split(":", 1)
+
+    if ("arch-" in device_arch):
+        env = copy.copy(llvm_config.config.environment)
+
+        # Find all available devices under the backend
+        env["ONEAPI_DEVICE_SELECTOR"] = backend + ":*"
+
+        # When using the ONEAPI_DEVICE_SELECTOR environment variable, sycl-ls
+        # prints warnings that might derail a user thinking something is wrong
+        # with their test run. It's just us filtering here, so silence them unless
+        # we get an exit status.
+        try:
+            cmd = "{} {} --verbose".format(config.run_launcher or "", sycl_ls)
+            sp = subprocess.run(cmd, env=env, text=True, shell=True, capture_output=True)
+            sp.check_returncode()
+        except subprocess.CalledProcessError as e:
+            # capturing e allows us to see path resolution errors / system
+            # permissions errors etc
+            lit_config.fatal(
+                f"Cannot list device architectures under {sycl_device}\n"
+                f"{e}\n"
+                f"stdout:{sp.stdout}\n"
+                f"stderr:{sp.stderr}\n"
+            )
+
+        detected_architectures = []
+
+        for line in sp.stdout.splitlines():
+            if re.match(r" *Architecture:", line):
+                _, architecture = line.strip().split(":", 1)
+                detected_architectures.append(architecture.strip())
+
+        device = device_arch.replace("arch-", "")
+
+        if device in detected_architectures:
+            device_num = detected_architectures.index(device)
+            filtered_sycl_devices.append(backend + ":" + str(device_num))
+        else:
+            lit_config.warning("Couldn't find device with architecture {}" \
+            " under {} device selector! Skipping device " \
+            "{}".format(device, backend + ":*", sycl_device))
path: sycl/test-e2e/lit.cfg.py
user: login: aelovikov-intel
body: Instead of this I'd suggest implementing
```
$ sycl-ls --show-architecture
[level_zero:gpu][level_zero:0] Name [Version]
  Architecture: whatever
...
```
(or maybe even single-line format, possibly ignoring name/version)

and simplifying this parsing. Not sure if that's too much to ask though.
created_at: 2025-05-08T21:07:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -821,6 +826,70 @@ def remove_level_zero_suffix(devices):
 if config.test_mode != "build-only":
     config.sycl_build_targets = set()
 
+# A device filter such as level_zero:gpu can have multiple devices under it and
+# the order is not guaranteed. The aspects enabled are also restricted to what
+# is supported on all devices under the label. It is possible for level_zero:gpu
+# and level_zero:0 to select different devices on different machines with the
+# same hardware. It is not currently possible to pass the device architecture to
+# ONEAPI_DEVICE_SELECTOR. Instead, if "BACKEND:arch-DEVICE_ARCH" is provided to
+# "sycl_devices", giving the desired device architecture, select a device that
+# matches that architecture using the backend:device-num device selection
+# scheme.
+filtered_sycl_devices = []
+for sycl_device in remove_level_zero_suffix(config.sycl_devices):
+
+    backend, device_arch  = sycl_device.split(":", 1)
+
+    if ("arch-" in device_arch):
path: sycl/test-e2e/lit.cfg.py
user: login: DBDuncan
body: I inverted and added early continue. Not used list comprehension much at all so quicker this way.
created_at: 2025-05-09T14:56:09Z
author_association: CONTRIBUTOR
diff_hunk: @@ -821,6 +826,70 @@ def remove_level_zero_suffix(devices):
 if config.test_mode != "build-only":
     config.sycl_build_targets = set()
 
+# A device filter such as level_zero:gpu can have multiple devices under it and
+# the order is not guaranteed. The aspects enabled are also restricted to what
+# is supported on all devices under the label. It is possible for level_zero:gpu
+# and level_zero:0 to select different devices on different machines with the
+# same hardware. It is not currently possible to pass the device architecture to
+# ONEAPI_DEVICE_SELECTOR. Instead, if "BACKEND:arch-DEVICE_ARCH" is provided to
+# "sycl_devices", giving the desired device architecture, select a device that
+# matches that architecture using the backend:device-num device selection
+# scheme.
+filtered_sycl_devices = []
+for sycl_device in remove_level_zero_suffix(config.sycl_devices):
+
+    backend, device_arch  = sycl_device.split(":", 1)
+
+    if ("arch-" in device_arch):
+        env = copy.copy(llvm_config.config.environment)
+
+        # Find all available devices under the backend
+        env["ONEAPI_DEVICE_SELECTOR"] = backend + ":*"
+
+        # When using the ONEAPI_DEVICE_SELECTOR environment variable, sycl-ls
+        # prints warnings that might derail a user thinking something is wrong
+        # with their test run. It's just us filtering here, so silence them unless
+        # we get an exit status.
path: sycl/test-e2e/lit.cfg.py
user: login: DBDuncan
body: Added helper function and added `with text_env()` good idea to help prevent any environment weirdness.
created_at: 2025-05-09T14:56:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -821,6 +826,70 @@ def remove_level_zero_suffix(devices):
 if config.test_mode != "build-only":
     config.sycl_build_targets = set()
 
+# A device filter such as level_zero:gpu can have multiple devices under it and
+# the order is not guaranteed. The aspects enabled are also restricted to what
+# is supported on all devices under the label. It is possible for level_zero:gpu
+# and level_zero:0 to select different devices on different machines with the
+# same hardware. It is not currently possible to pass the device architecture to
+# ONEAPI_DEVICE_SELECTOR. Instead, if "BACKEND:arch-DEVICE_ARCH" is provided to
+# "sycl_devices", giving the desired device architecture, select a device that
+# matches that architecture using the backend:device-num device selection
+# scheme.
+filtered_sycl_devices = []
+for sycl_device in remove_level_zero_suffix(config.sycl_devices):
+
+    backend, device_arch  = sycl_device.split(":", 1)
+
+    if ("arch-" in device_arch):
+        env = copy.copy(llvm_config.config.environment)
+
+        # Find all available devices under the backend
+        env["ONEAPI_DEVICE_SELECTOR"] = backend + ":*"
+
+        # When using the ONEAPI_DEVICE_SELECTOR environment variable, sycl-ls
+        # prints warnings that might derail a user thinking something is wrong
+        # with their test run. It's just us filtering here, so silence them unless
+        # we get an exit status.
+        try:
+            cmd = "{} {} --verbose".format(config.run_launcher or "", sycl_ls)
+            sp = subprocess.run(cmd, env=env, text=True, shell=True, capture_output=True)
+            sp.check_returncode()
+        except subprocess.CalledProcessError as e:
+            # capturing e allows us to see path resolution errors / system
+            # permissions errors etc
+            lit_config.fatal(
+                f"Cannot list device architectures under {sycl_device}\n"
+                f"{e}\n"
+                f"stdout:{sp.stdout}\n"
+                f"stderr:{sp.stderr}\n"
+            )
+
+        detected_architectures = []
+
+        for line in sp.stdout.splitlines():
+            if re.match(r" *Architecture:", line):
+                _, architecture = line.strip().split(":", 1)
+                detected_architectures.append(architecture.strip())
+
+        device = device_arch.replace("arch-", "")
+
+        if device in detected_architectures:
+            device_num = detected_architectures.index(device)
+            filtered_sycl_devices.append(backend + ":" + str(device_num))
+        else:
+            lit_config.warning("Couldn't find device with architecture {}" \
+            " under {} device selector! Skipping device " \
+            "{}".format(device, backend + ":*", sycl_device))
path: sycl/test-e2e/lit.cfg.py
user: login: DBDuncan
body: Not sure. I feel it would feel weird implementing that only for one field and not the others. Plus if something like that was done for all fields that would prob be too much. I think it might make more sense to do this if `sycl-ls` and `ONEAPI_DEVICE_SELECTOR` are made to accept `level_zero:arch-intel_gpu_bmg_g21`.
created_at: 2025-05-09T14:56:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -828,6 +833,73 @@ def remove_level_zero_suffix(devices):
 if config.test_mode != "build-only":
     config.sycl_build_targets = set()
 
+
+def get_sycl_ls_verbose(sycl_device, env):
+    with test_env():
+        # When using the ONEAPI_DEVICE_SELECTOR environment variable, sycl-ls
+        # prints warnings that might derail a user thinking something is wrong
+        # with their test run. It's just us filtering here, so silence them unless
+        # we get an exit status.
+        try:
+            cmd = "{} {} --verbose".format(config.run_launcher or "", sycl_ls)
+            sp = subprocess.run(cmd, env=env, text=True, shell=True, capture_output=True)
+            sp.check_returncode()
+        except subprocess.CalledProcessError as e:
+            # capturing e allows us to see path resolution errors / system
+            # permissions errors etc
+            lit_config.fatal(
+                f"Cannot find devices under {sycl_device}\n"
+                f"{e}\n"
+                f"stdout:{sp.stdout}\n"
+                f"stderr:{sp.stderr}\n"
+            )
+        return sp.stdout.splitlines()
+
+# A device filter such as level_zero:gpu can have multiple devices under it and
+# the order is not guaranteed. The aspects enabled are also restricted to what
+# is supported on all devices under the label. It is possible for level_zero:gpu
+# and level_zero:0 to select different devices on different machines with the
+# same hardware. It is not currently possible to pass the device architecture to
+# ONEAPI_DEVICE_SELECTOR. Instead, if "BACKEND:arch-DEVICE_ARCH" is provided to
+# "sycl_devices", giving the desired device architecture, select a device that
+# matches that architecture using the backend:device-num device selection
+# scheme.
+filtered_sycl_devices = []
+for sycl_device in remove_level_zero_suffix(config.sycl_devices):
+    backend, device_arch  = sycl_device.split(":", 1)
+
+    if not "arch-" in device_arch:
+        filtered_sycl_devices.append(sycl_device)
+        continue
+
+    env = copy.copy(llvm_config.config.environment)
+
+    # Find all available devices under the backend
+    env["ONEAPI_DEVICE_SELECTOR"] = backend + ":*"
+
+    detected_architectures = []
+
+    for line in get_sycl_ls_verbose(backend + ":*", env):
path: sycl/test-e2e/lit.cfg.py
user: login: sarnex
body: i really double this actually matters but is it possible to prevent a second call to `sycl-ls` here and capture what we need in the first call?
created_at: 2025-05-09T15:23:34Z
author_association: CONTRIBUTOR
diff_hunk: @@ -828,6 +833,73 @@ def remove_level_zero_suffix(devices):
 if config.test_mode != "build-only":
     config.sycl_build_targets = set()
 
+
+def get_sycl_ls_verbose(sycl_device, env):
+    with test_env():
+        # When using the ONEAPI_DEVICE_SELECTOR environment variable, sycl-ls
+        # prints warnings that might derail a user thinking something is wrong
+        # with their test run. It's just us filtering here, so silence them unless
+        # we get an exit status.
+        try:
+            cmd = "{} {} --verbose".format(config.run_launcher or "", sycl_ls)
+            sp = subprocess.run(cmd, env=env, text=True, shell=True, capture_output=True)
+            sp.check_returncode()
+        except subprocess.CalledProcessError as e:
+            # capturing e allows us to see path resolution errors / system
+            # permissions errors etc
+            lit_config.fatal(
+                f"Cannot find devices under {sycl_device}\n"
+                f"{e}\n"
+                f"stdout:{sp.stdout}\n"
+                f"stderr:{sp.stderr}\n"
+            )
+        return sp.stdout.splitlines()
+
+# A device filter such as level_zero:gpu can have multiple devices under it and
+# the order is not guaranteed. The aspects enabled are also restricted to what
+# is supported on all devices under the label. It is possible for level_zero:gpu
+# and level_zero:0 to select different devices on different machines with the
+# same hardware. It is not currently possible to pass the device architecture to
+# ONEAPI_DEVICE_SELECTOR. Instead, if "BACKEND:arch-DEVICE_ARCH" is provided to
+# "sycl_devices", giving the desired device architecture, select a device that
+# matches that architecture using the backend:device-num device selection
+# scheme.
+filtered_sycl_devices = []
+for sycl_device in remove_level_zero_suffix(config.sycl_devices):
+    backend, device_arch  = sycl_device.split(":", 1)
+
+    if not "arch-" in device_arch:
+        filtered_sycl_devices.append(sycl_device)
+        continue
+
+    env = copy.copy(llvm_config.config.environment)
+
+    # Find all available devices under the backend
+    env["ONEAPI_DEVICE_SELECTOR"] = backend + ":*"
+
+    detected_architectures = []
+
+    for line in get_sycl_ls_verbose(backend + ":*", env):
path: sycl/test-e2e/lit.cfg.py
user: login: aelovikov-intel
body: Don't we use different device filter?
created_at: 2025-05-09T15:27:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -828,6 +833,73 @@ def remove_level_zero_suffix(devices):
 if config.test_mode != "build-only":
     config.sycl_build_targets = set()
 
+
+def get_sycl_ls_verbose(sycl_device, env):
+    with test_env():
+        # When using the ONEAPI_DEVICE_SELECTOR environment variable, sycl-ls
+        # prints warnings that might derail a user thinking something is wrong
+        # with their test run. It's just us filtering here, so silence them unless
+        # we get an exit status.
+        try:
+            cmd = "{} {} --verbose".format(config.run_launcher or "", sycl_ls)
+            sp = subprocess.run(cmd, env=env, text=True, shell=True, capture_output=True)
+            sp.check_returncode()
+        except subprocess.CalledProcessError as e:
+            # capturing e allows us to see path resolution errors / system
+            # permissions errors etc
+            lit_config.fatal(
+                f"Cannot find devices under {sycl_device}\n"
+                f"{e}\n"
+                f"stdout:{sp.stdout}\n"
+                f"stderr:{sp.stderr}\n"
+            )
+        return sp.stdout.splitlines()
+
+# A device filter such as level_zero:gpu can have multiple devices under it and
+# the order is not guaranteed. The aspects enabled are also restricted to what
+# is supported on all devices under the label. It is possible for level_zero:gpu
+# and level_zero:0 to select different devices on different machines with the
+# same hardware. It is not currently possible to pass the device architecture to
+# ONEAPI_DEVICE_SELECTOR. Instead, if "BACKEND:arch-DEVICE_ARCH" is provided to
+# "sycl_devices", giving the desired device architecture, select a device that
+# matches that architecture using the backend:device-num device selection
+# scheme.
+filtered_sycl_devices = []
+for sycl_device in remove_level_zero_suffix(config.sycl_devices):
+    backend, device_arch  = sycl_device.split(":", 1)
+
+    if not "arch-" in device_arch:
+        filtered_sycl_devices.append(sycl_device)
+        continue
+
+    env = copy.copy(llvm_config.config.environment)
+
+    # Find all available devices under the backend
+    env["ONEAPI_DEVICE_SELECTOR"] = backend + ":*"
+
+    detected_architectures = []
+
+    for line in get_sycl_ls_verbose(backend + ":*", env):
path: sycl/test-e2e/lit.cfg.py
user: login: sarnex
body: yeah in this one we set a filter but it seems the other one has no device filter, so it's output should contain the info we need here, but if its not trivial to implement its not worth it
created_at: 2025-05-09T15:33:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -828,6 +833,73 @@ def remove_level_zero_suffix(devices):
 if config.test_mode != "build-only":
     config.sycl_build_targets = set()
 
+
+def get_sycl_ls_verbose(sycl_device, env):
+    with test_env():
+        # When using the ONEAPI_DEVICE_SELECTOR environment variable, sycl-ls
+        # prints warnings that might derail a user thinking something is wrong
+        # with their test run. It's just us filtering here, so silence them unless
+        # we get an exit status.
+        try:
+            cmd = "{} {} --verbose".format(config.run_launcher or "", sycl_ls)
+            sp = subprocess.run(cmd, env=env, text=True, shell=True, capture_output=True)
+            sp.check_returncode()
+        except subprocess.CalledProcessError as e:
+            # capturing e allows us to see path resolution errors / system
+            # permissions errors etc
+            lit_config.fatal(
+                f"Cannot find devices under {sycl_device}\n"
+                f"{e}\n"
+                f"stdout:{sp.stdout}\n"
+                f"stderr:{sp.stderr}\n"
+            )
+        return sp.stdout.splitlines()
+
+# A device filter such as level_zero:gpu can have multiple devices under it and
+# the order is not guaranteed. The aspects enabled are also restricted to what
+# is supported on all devices under the label. It is possible for level_zero:gpu
+# and level_zero:0 to select different devices on different machines with the
+# same hardware. It is not currently possible to pass the device architecture to
+# ONEAPI_DEVICE_SELECTOR. Instead, if "BACKEND:arch-DEVICE_ARCH" is provided to
+# "sycl_devices", giving the desired device architecture, select a device that
+# matches that architecture using the backend:device-num device selection
+# scheme.
+filtered_sycl_devices = []
+for sycl_device in remove_level_zero_suffix(config.sycl_devices):
+    backend, device_arch  = sycl_device.split(":", 1)
+
+    if not "arch-" in device_arch:
+        filtered_sycl_devices.append(sycl_device)
+        continue
+
+    env = copy.copy(llvm_config.config.environment)
+
+    # Find all available devices under the backend
+    env["ONEAPI_DEVICE_SELECTOR"] = backend + ":*"
+
+    detected_architectures = []
+
+    for line in get_sycl_ls_verbose(backend + ":*", env):
path: sycl/test-e2e/lit.cfg.py
user: login: DBDuncan
body: The first sycl-ls, line 638 in the code is used to get all available devices if `all` is used so no filter is passed.

The second sycl-ls, line 882 is only called if the string `arch-` appears in the device filter. We could also grab the aspects at the same time but that feels messy. 

Lastly, sycl-ls at line 938 is called again with all the filtered device filters to set available aspects and other properties.
created_at: 2025-05-09T15:36:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -828,6 +833,73 @@ def remove_level_zero_suffix(devices):
 if config.test_mode != "build-only":
     config.sycl_build_targets = set()
 
+
+def get_sycl_ls_verbose(sycl_device, env):
+    with test_env():
+        # When using the ONEAPI_DEVICE_SELECTOR environment variable, sycl-ls
+        # prints warnings that might derail a user thinking something is wrong
+        # with their test run. It's just us filtering here, so silence them unless
+        # we get an exit status.
+        try:
+            cmd = "{} {} --verbose".format(config.run_launcher or "", sycl_ls)
+            sp = subprocess.run(cmd, env=env, text=True, shell=True, capture_output=True)
+            sp.check_returncode()
+        except subprocess.CalledProcessError as e:
+            # capturing e allows us to see path resolution errors / system
+            # permissions errors etc
+            lit_config.fatal(
+                f"Cannot find devices under {sycl_device}\n"
+                f"{e}\n"
+                f"stdout:{sp.stdout}\n"
+                f"stderr:{sp.stderr}\n"
+            )
+        return sp.stdout.splitlines()
+
+# A device filter such as level_zero:gpu can have multiple devices under it and
+# the order is not guaranteed. The aspects enabled are also restricted to what
+# is supported on all devices under the label. It is possible for level_zero:gpu
+# and level_zero:0 to select different devices on different machines with the
+# same hardware. It is not currently possible to pass the device architecture to
+# ONEAPI_DEVICE_SELECTOR. Instead, if "BACKEND:arch-DEVICE_ARCH" is provided to
+# "sycl_devices", giving the desired device architecture, select a device that
+# matches that architecture using the backend:device-num device selection
+# scheme.
+filtered_sycl_devices = []
+for sycl_device in remove_level_zero_suffix(config.sycl_devices):
+    backend, device_arch  = sycl_device.split(":", 1)
+
+    if not "arch-" in device_arch:
+        filtered_sycl_devices.append(sycl_device)
+        continue
+
+    env = copy.copy(llvm_config.config.environment)
+
+    # Find all available devices under the backend
+    env["ONEAPI_DEVICE_SELECTOR"] = backend + ":*"
+
+    detected_architectures = []
+
+    for line in get_sycl_ls_verbose(backend + ":*", env):
path: sycl/test-e2e/lit.cfg.py
user: login: DBDuncan
body: We prob could remove one of the calls to sycl-ls in the code but I think the separation of concerns with filtering the device filters and then using them to query sycl-ls again to set test parameters makes things less tangled.
created_at: 2025-05-09T15:37:49Z
author_association: CONTRIBUTOR
diff_hunk: @@ -828,6 +833,73 @@ def remove_level_zero_suffix(devices):
 if config.test_mode != "build-only":
     config.sycl_build_targets = set()
 
+
+def get_sycl_ls_verbose(sycl_device, env):
+    with test_env():
+        # When using the ONEAPI_DEVICE_SELECTOR environment variable, sycl-ls
+        # prints warnings that might derail a user thinking something is wrong
+        # with their test run. It's just us filtering here, so silence them unless
+        # we get an exit status.
+        try:
+            cmd = "{} {} --verbose".format(config.run_launcher or "", sycl_ls)
+            sp = subprocess.run(cmd, env=env, text=True, shell=True, capture_output=True)
+            sp.check_returncode()
+        except subprocess.CalledProcessError as e:
+            # capturing e allows us to see path resolution errors / system
+            # permissions errors etc
+            lit_config.fatal(
+                f"Cannot find devices under {sycl_device}\n"
+                f"{e}\n"
+                f"stdout:{sp.stdout}\n"
+                f"stderr:{sp.stderr}\n"
+            )
+        return sp.stdout.splitlines()
+
+# A device filter such as level_zero:gpu can have multiple devices under it and
+# the order is not guaranteed. The aspects enabled are also restricted to what
+# is supported on all devices under the label. It is possible for level_zero:gpu
+# and level_zero:0 to select different devices on different machines with the
+# same hardware. It is not currently possible to pass the device architecture to
+# ONEAPI_DEVICE_SELECTOR. Instead, if "BACKEND:arch-DEVICE_ARCH" is provided to
+# "sycl_devices", giving the desired device architecture, select a device that
+# matches that architecture using the backend:device-num device selection
+# scheme.
+filtered_sycl_devices = []
+for sycl_device in remove_level_zero_suffix(config.sycl_devices):
+    backend, device_arch  = sycl_device.split(":", 1)
+
+    if not "arch-" in device_arch:
+        filtered_sycl_devices.append(sycl_device)
+        continue
+
+    env = copy.copy(llvm_config.config.environment)
+
+    # Find all available devices under the backend
+    env["ONEAPI_DEVICE_SELECTOR"] = backend + ":*"
+
+    detected_architectures = []
+
+    for line in get_sycl_ls_verbose(backend + ":*", env):
path: sycl/test-e2e/lit.cfg.py
user: login: sarnex
body: sure, thanks
created_at: 2025-05-09T15:42:45Z
author_association: CONTRIBUTOR

PR comments:
user: login: aelovikov-intel
created_at: 2025-04-25T15:43:33Z
body: What problem are you trying to solve? Empty PR description and no comments in the change don't help to understand the idea behind this.
author_association: CONTRIBUTOR
user: login: DBDuncan
created_at: 2025-04-25T17:30:40Z
body: Ah sorry. You are right. The issue is that on devices with say a battlemage GPU and an intel intergrated iGPU, when running llvm-lit with `--param sycl_devices="level_zero:gpu"` set, when sycl/test-e2e/lit.cfg.py is sorting out which aspects are enabled, it only signals aspects as supported to `// REQUIRES: aspect-` that are supported on all intel devices listed via `level_zero:gpu` from sycl-ls.

This is a problem with bindless images as the aspect `exp_oneapi_bindless_images` is not supported on intel iGPUs, but even if the tests are running on a supported device because that device comes first in the list it will not run them because the aspect in tests returns the feature as unsupported.

Currently, sycl/test-e2e/lit.cfg.py does not support the `level_zero:0/1/2/etc` device selection style which would fix this issue. So patch intends to fix this allowing for explicit selection for the device, not just platform and device type.
author_association: CONTRIBUTOR
user: login: aelovikov-intel
created_at: 2025-04-25T17:36:59Z
body: First, that should go to the PR description, not the comment. Second, please add an example usage, e.g., sycl-ls shows this, we run llvm-lit like that, and it does the following (and print detected aspects from its output).
author_association: CONTRIBUTOR
user: login: DBDuncan
created_at: 2025-05-02T15:41:58Z
body: I am putting this as a comment here as it feels like a lot to put in the commit message. Though I can put it there if you prefer or a cut down version. I might be going a bit too verbose.

On the system I am using, when running sycl-ls like this it shows these devices as available:

./bin/sycl-ls`

```
[level_zero:gpu][level_zero:0] Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) Arc(TM) B580 Graphics 20.1.0 [1.6.32567+16]
[level_zero:gpu][level_zero:1] Intel(R) oneAPI Unified Runtime over Level-Zero, Intel(R) UHD Graphics 770 12.2.0 [1.6.32567+16]
[opencl:gpu][opencl:0] Intel(R) OpenCL Graphics, Intel(R) Arc(TM) B580 Graphics OpenCL 3.0 NEO  [25.05.32567]
[opencl:gpu][opencl:1] Intel(R) OpenCL Graphics, Intel(R) UHD Graphics 770 OpenCL 3.0 NEO  [25.05.32567]
```

I am abbreviating this, but adding `--verbose` shows the following aspects as available for the battlemage:

```
        Aspects           : gpu fp16 fp64 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_oneapi_cuda_async_barrier ext_intel_free_memory ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_intel_legacy_image ext_oneapi_bindless_images ext_oneapi_bindless_images_1d_usm ext_oneapi_bindless_images_2d_usm ext_oneapi_external_memory_import ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_intel_matrix ext_oneapi_limited_graph ext_oneapi_private_alloca ext_oneapi_bindless_sampled_image_fetch_1d_usm ext_oneapi_bindless_sampled_image_fetch_2d_usm ext_oneapi_bindless_sampled_image_fetch_2d ext_oneapi_bindless_sampled_image_fetch_3d ext_oneapi_queue_profiling_tag ext_oneapi_virtual_mem ext_oneapi_image_array ext_oneapi_virtual_functions ext_intel_spill_memory_size ext_intel_current_clock_throttle_reasons ext_intel_power_limits ext_oneapi_async_memory_alloc
```

And the intel Intel(R) UHD Graphics 770:

```
        Aspects           : gpu fp16 online_compiler online_linker queue_profiling usm_device_allocations usm_host_allocations usm_shared_allocations ext_intel_pci_address ext_intel_gpu_eu_count ext_intel_gpu_eu_simd_width ext_intel_gpu_slices ext_intel_gpu_subslices_per_slice ext_intel_gpu_eu_count_per_subslice atomic64 ext_intel_device_info_uuid ext_intel_gpu_hw_threads_per_eu ext_oneapi_cuda_async_barrier ext_intel_device_id ext_intel_memory_clock_rate ext_intel_memory_bus_width ext_intel_legacy_image ext_oneapi_external_memory_import ext_intel_esimd ext_oneapi_ballot_group ext_oneapi_fixed_size_group ext_oneapi_opportunistic_group ext_oneapi_tangle_group ext_oneapi_limited_graph ext_oneapi_private_alloca ext_oneapi_bindless_sampled_image_fetch_1d_usm ext_oneapi_bindless_sampled_image_fetch_2d_usm ext_oneapi_bindless_sampled_image_fetch_2d ext_oneapi_bindless_sampled_image_fetch_3d ext_oneapi_queue_profiling_tag ext_oneapi_virtual_mem ext_oneapi_image_array ext_oneapi_virtual_functions ext_intel_spill_memory_size ext_intel_current_clock_throttle_reasons ext_intel_power_limits ext_oneapi_async_memory_alloc
```

When running llvm-lit via this command:
`python3.12 bin/llvm-lit -sva --param sycl_devices="level_zero:gpu" ../sycl/test-e2e/bindless_images/`

I get the following output before the tests start running:

```
llvm-lit: /home/duncanb/git/llvm/sycl/test-e2e/lit.cfg.py:613: note: Targeted devices: level_zero:gpu
llvm-lit: /home/duncanb/git/llvm/sycl/test-e2e/lit.cfg.py:815: note: Found pre-installed AOT device compiler ocloc
llvm-lit: /home/duncanb/git/llvm/sycl/test-e2e/lit.cfg.py:818: warning: Couldn't find pre-installed AOT device compiler opencl-aot
llvm-lit: /home/duncanb/git/llvm/sycl/test-e2e/lit.cfg.py:929: note: Aspects for level_zero:gpu: ext_intel_power_limits, ext_intel_current_clock_throttle_reasons, ext_oneapi_private_alloca, ext_intel_gpu_eu_count, ext_oneapi_virtual_mem, atomic64, queue_profiling, ext_oneapi_queue_profiling_tag, ext_oneapi_external_memory_import, ext_intel_device_info_uuid, ext_intel_pci_address, usm_host_allocations, ext_oneapi_virtual_functions, ext_oneapi_limited_graph, ext_oneapi_ballot_group, online_compiler, ext_intel_legacy_image, ext_oneapi_opportunistic_group, ext_oneapi_bindless_sampled_image_fetch_1d_usm, ext_intel_gpu_hw_threads_per_eu, fp16, ext_oneapi_bindless_sampled_image_fetch_2d_usm, usm_shared_allocations, ext_intel_memory_bus_width, ext_oneapi_fixed_size_group, ext_intel_gpu_eu_simd_width, ext_intel_gpu_subslices_per_slice, online_linker, ext_intel_spill_memory_size, ext_oneapi_async_memory_alloc, ext_oneapi_bindless_sampled_image_fetch_2d, ext_intel_esimd, ext_oneapi_cuda_async_barrier, gpu, usm_device_allocations, ext_intel_memory_clock_rate, ext_intel_device_id, ext_oneapi_image_array, ext_intel_gpu_eu_count_per_subslice, ext_oneapi_bindless_sampled_image_fetch_3d, ext_oneapi_tangle_group, ext_intel_gpu_slices
llvm-lit: /home/duncanb/git/llvm/sycl/test-e2e/lit.cfg.py:941: note: SG sizes for level_zero:gpu: 32, 16
llvm-lit: /home/duncanb/git/llvm/sycl/test-e2e/lit.cfg.py:950: note: Architectures for level_zero:gpu: intel_gpu_adl_s, intel_gpu_bmg_g21
```

Then, running llvm-lit using `level_zero:0` like this:
`python3.12 bin/llvm-lit -sva --param sycl_devices="level_zero:0" ../sycl/test-e2e/bindless_images/`

Get the following output:

```
llvm-lit: /home/duncanb/git/llvm/sycl/test-e2e/lit.cfg.py:613: note: Targeted devices: level_zero:0
llvm-lit: /home/duncanb/git/llvm/sycl/test-e2e/lit.cfg.py:815: note: Found pre-installed AOT device compiler ocloc
llvm-lit: /home/duncanb/git/llvm/sycl/test-e2e/lit.cfg.py:818: warning: Couldn't find pre-installed AOT device compiler opencl-aot
llvm-lit: /home/duncanb/git/llvm/sycl/test-e2e/lit.cfg.py:929: note: Aspects for level_zero:0: ext_oneapi_tangle_group, ext_intel_legacy_image, ext_intel_gpu_slices, ext_oneapi_bindless_sampled_image_fetch_2d, ext_intel_power_limits, ext_oneapi_async_memory_alloc, ext_oneapi_private_alloca, ext_intel_matrix, usm_shared_allocations, online_linker, usm_host_allocations, ext_intel_memory_bus_width, ext_oneapi_external_memory_import, ext_oneapi_bindless_sampled_image_fetch_1d_usm, ext_oneapi_queue_profiling_tag, online_compiler, ext_oneapi_virtual_mem, usm_device_allocations, ext_oneapi_image_array, ext_intel_gpu_eu_count_per_subslice, ext_intel_memory_clock_rate, ext_oneapi_bindless_sampled_image_fetch_3d, ext_intel_spill_memory_size, ext_oneapi_ballot_group, ext_oneapi_cuda_async_barrier, ext_intel_gpu_eu_simd_width, ext_intel_gpu_subslices_per_slice, ext_intel_device_id, ext_oneapi_bindless_images, ext_oneapi_bindless_images_2d_usm, ext_intel_gpu_eu_count, queue_profiling, ext_oneapi_virtual_functions, fp64, ext_oneapi_fixed_size_group, ext_intel_device_info_uuid, ext_oneapi_bindless_sampled_image_fetch_2d_usm, ext_intel_pci_address, ext_oneapi_bindless_images_1d_usm, fp16, ext_intel_free_memory, ext_intel_current_clock_throttle_reasons, ext_intel_esimd, gpu, ext_oneapi_opportunistic_group, ext_oneapi_limited_graph, atomic64, ext_intel_gpu_hw_threads_per_eu
llvm-lit: /home/duncanb/git/llvm/sycl/test-e2e/lit.cfg.py:941: note: SG sizes for level_zero:0: 32, 16
llvm-lit: /home/duncanb/git/llvm/sycl/test-e2e/lit.cfg.py:950: note: Architectures for level_zero:0: intel_gpu_bmg_g21
```

As can be seen with the llvm-lit command using `level_zero:gpu`, the aspect `ext_oneapi_bindless_images` is not available but when using the more specific device selection style of `level_zero:0 it is.

As far as I am aware, it is not currently possible to make llvm-lit sycl run tests that require an aspect that is available on a device selectable via `level_zero:gpu` but not another that shows under the same label on the same system without making changes to the code.

Enabling the `level_zero:0` device selection style via llvm-lit should make it easier to run tests on systems with multiple GPUs without needing to disable one GPU or force enable the aspect.
author_association: CONTRIBUTOR
user: login: DBDuncan
created_at: 2025-05-02T15:42:19Z
body: @aelovikov-intel I apologize for taking so long. Other work got in the way.

I might have gone overboard with all the aspects I am showing but I felt I would be leaving out context without them. I appreciate your feedback if you want anything else.
author_association: CONTRIBUTOR
user: login: aelovikov-intel
created_at: 2025-05-02T15:55:06Z
body: I don't like `--param sycl_devices="level_zero:0"` for CI purposes because it will be possible to silently loose coverage if devices order changes for some reason. IMO, a better approach is to somehow pass `intel_gpu_bmg_g21` as the intended target. I'm not sure how to do that though.

I'd like to see the following logic: if we have multiple `backend:device_type` devices then extra parameter `--param require-device-arch=<value>` is required. Then, the `lit.cfg.py` should verify that only one of `backend:device_type` devices satisfy that condition and select that number internally.

However, that won't work with multi-tile GPUs that are seen as separate root devices. Maybe "only one" above is unnecessary and we should always select a numbered device out of several options matching a given `backend:device_type`. The filtering via `--param require-device-arch=<value>` above should still take place.

WDYT? \+ @sarnex , @uditagarwal97 
author_association: CONTRIBUTOR
user: login: sarnex
created_at: 2025-05-02T18:31:38Z
body: I agree, I don't like using the device ID number, totally agree we could silently loose coverage.

I like the `require-device-arch` idea, but ideally `ONEAPI_DEVICE_SELECTOR` would support `level_zero:intel_gpu_bmg_g21` or something so the SYCL runtime would have to figure it out instead of the LIT code.

But since we don't have that, a LIT param is fine.

For the multi-tile case, I'm sure we can figure it out. The devices will probably have the same UUID or PCI bus address or something so we can detect it's multi-tile.
author_association: CONTRIBUTOR
user: login: DBDuncan
created_at: 2025-05-05T14:53:42Z
body: This seems like a good solution. The only thing is because `sycl_devices` accepts multiple parameters, `require-device-arch` likely should as well.

I am thinking to not require to put as many `require-device-arch` as there are `sycl_devices`, and instead perhaps emit a warning if the number does not match up and then default to the current behaviour. Any thoughts?
author_association: CONTRIBUTOR
user: login: sarnex
created_at: 2025-05-05T14:58:52Z
body: I think we can do a sensible simple default and then expand it in the future if we need it. Maybe we could just have a way to specify `cpu` and `gpu` devices so that we could run `opencl:gpu` and `opencl:gpu` testing at the same time but still pass `require-device-arch`.

Maybe something like

``--param require-device-arch="gpu:intel_gpu_bmg_g21;cpu:sapphire_rapids"`` 
(I have no idea what the arches are for CPU I just made something up

and we would apply the `gpu` specification to both `level_zero:gpu` and `opencl:gpu` but not `opencl:cpu` and vice versa.
author_association: CONTRIBUTOR
user: login: aelovikov-intel
created_at: 2025-05-05T15:05:34Z
body: I think CPU arch isn't useful. Can you imagine a system with two different CPUs on it? We aren't MPI...
author_association: CONTRIBUTOR
user: login: sarnex
created_at: 2025-05-05T15:10:50Z
body: Yeah I don't have a strong opinion, for now we could only support GPUs. Maybe name the option `require-gpu-device-arch` to make it clear? Important thing is to apply it for both `level_zero:gpu` and `opencl:gpu` and `CUDA/HIP` tho
author_association: CONTRIBUTOR
user: login: DBDuncan
created_at: 2025-05-06T13:49:38Z
body: I have been investigating implementing this and I believe internally I will need to make it so that when `--param sycl_devices="level_zero:gpu" --param require-gpu-device-arch="intel_gpu_bmg_g21` is set it will need to internally swap `level_zero:gpu` with the appropriate `level_zero:N` device selector as there seems to be no other way to specificity select a device.
author_association: CONTRIBUTOR
user: login: aelovikov-intel
created_at: 2025-05-06T14:20:33Z
body: > I have been investigating implementing this and I believe internally I will need to make it so that when `--param sycl_devices="level_zero:gpu" --param require-gpu-device-arch="intel_gpu_bmg_g21` is set it will need to internally swap `level_zero:gpu` with the appropriate `level_zero:N` device selector as there seems to be no other way to specificity select a device.

That's what I was thinking. Maybe even go further and do a switch anytime there are multiple devices matching `backend:device-type`.
author_association: CONTRIBUTOR
user: login: DBDuncan
created_at: 2025-05-08T13:12:18Z
body: @sarnex @aelovikov-intel Here is the new version using `require-device-arch`. Behaviour should be the same as before if `require-device-arch` is not used. Bit uncertain how to go about CI testing.

I designed it so you can only filter devices on a one to one basis. Meaning that if say you want to run both `intel_gpu_adl_s` and `intel_gpu_bmg_g21` which both fall under the `level_zero:gpu label, you would have to do this: `--param sycl_devices="level_zero:gpu;level_zero:gpu` --param require_device_arch="intel_gpu_bmg_g21;intel_gpu_adl_s"

Feedback is appreciated! This design is perhaps slightly different then the design discussed in this thread before but should be a good starting point.

Apologies if the python code is not very pythonic. Have not written in it for a while but is nice to get some more experience.
author_association: CONTRIBUTOR
user: login: aelovikov-intel
created_at: 2025-05-08T14:11:11Z
body: Brainstorming... What if we do it like this: `--param sycl_devices=level_zero:arch-intel_gpu_bmg_g21` ?
author_association: CONTRIBUTOR
user: login: sarnex
created_at: 2025-05-08T14:15:03Z
body: yeah i like andrei's idea, today with `sycl_devices=level_zero:gpu` we would only ever be testing one device so we ideally wouldnt change that with this new param, so if we can specify the device multiple times with a specific arch that would solve it
author_association: CONTRIBUTOR
user: login: DBDuncan
created_at: 2025-05-08T14:15:57Z
body: Ooh, that actually could work. Internally would have to check for devices via `level_zero:*` and then filter based upon those results.
author_association: CONTRIBUTOR
user: login: DBDuncan
created_at: 2025-05-08T14:19:08Z
body: I think I would have to make it so that if the second part of  `level_zero:arch-intel_gpu_bmg_g21 ` is not recognised based upon

```python
available_devices = {
    "opencl": ("cpu", "gpu", "fpga"),
    "cuda": "gpu",
    "level_zero": "gpu",
    "hip": "gpu",
    "native_cpu": "cpu",
}
```

then assume it is a device architecture specific filter.
author_association: CONTRIBUTOR
user: login: aelovikov-intel
created_at: 2025-05-08T14:20:06Z
body: My assumption was that `arch-` prefix can serve as the trigger for this logic.
author_association: CONTRIBUTOR
user: login: DBDuncan
created_at: 2025-05-08T14:20:32Z
body: I am blind. Yep that should be fine.
author_association: CONTRIBUTOR
user: login: DBDuncan
created_at: 2025-05-08T17:14:03Z
body: Updated. This version is much more ergonomic. Can pass `--param sycl_devices="level_zero:arch-intel_gpu_bmg_g21"
author_association: CONTRIBUTOR
user: login: DBDuncan
created_at: 2025-05-09T15:13:15Z
body: I missed a third `sycl-ls` invocation. Slightly different this one as it does not call `--verbose`. Just modified it to add `with test_env():`.
author_association: CONTRIBUTOR
user: login: sarnex
created_at: 2025-05-09T15:43:11Z
body: Once you fix the code formatting CI fail we can merge this IMO
author_association: CONTRIBUTOR
user: login: DBDuncan
created_at: 2025-05-09T15:53:33Z
body: > Once you fix the code formatting CI fail we can merge this IMO

I knew I forgot something. Had to install the black formatter extension. Should be good now.
author_association: CONTRIBUTOR
user: login: sarnex
created_at: 2025-05-09T15:57:32Z
body: thanks, this is a really useful feature!
author_association: CONTRIBUTOR
user: login: DBDuncan
created_at: 2025-05-09T16:02:33Z
body: Thanks! There is a very minor documentation update i missed at line 281 of sycl/test-e2e/README.md just to explain that device selection like  `level_zero:arch-intel_gpu_bmg_g21` is accepted. Like I pushed the fix just 2 mins after this was merged lol. Coming to the end of the day and is a very minor change. Will make a PR correcting this on Monday.
author_association: CONTRIBUTOR
user: login: sarnex
created_at: 2025-05-09T16:04:19Z
body: Np, thanks, take it easy!
author_association: CONTRIBUTOR
user: login: hvdijk
created_at: 2025-05-16T12:34:11Z
body: The refactor to call `sp = subprocess.Run(...)` in a function means that later on, in e.g.
```
        lit_config.error(
            "Cannot detect device aspect for {}\nstdout:\n{}\nstderr:\n{}".format(
                sycl_device, sp.stdout, sp.stderr
            )
        )
```
`sp` is no longer available and this crashes rather than reporting an appropriate error. Likewise for other `lit_config.error`s in the same file. Could you take a look? I can open a new issue for this if you prefer?
author_association: CONTRIBUTOR
user: login: DBDuncan
created_at: 2025-05-19T12:45:55Z
body: > The refactor to call `sp = subprocess.Run(...)` in a function means that later on, in e.g.
> 
> ```
>         lit_config.error(
>             "Cannot detect device aspect for {}\nstdout:\n{}\nstderr:\n{}".format(
>                 sycl_device, sp.stdout, sp.stderr
>             )
>         )
> ```
> 
> `sp` is no longer available and this crashes rather than reporting an appropriate error. Likewise for other `lit_config.error`s in the same file. Could you take a look? I can open a new issue for this if you prefer?

Sorry Harald, just got back from holiday. Will fix this. Need to be more careful jumping from a compiled to interpreted language with things like this...
author_association: CONTRIBUTOR
user: login: hvdijk
created_at: 2025-05-19T12:47:08Z
body: @RossBrunton also separately encountered this and already put up a fix as #18535 🙂 
author_association: CONTRIBUTOR
user: login: DBDuncan
created_at: 2025-05-19T12:47:28Z
body: Ah, ok lol
author_association: CONTRIBUTOR