Repository: bazelbuild/bazel
Issue: Provide the host CPU to repository rules
Labels: type: feature request
P2
team-ExternalDeps
help wanted

<!--
ATTENTION! Please read and follow:
- if this is a _question_ about how to build / test / query / deploy using Bazel, or a _discussion starter_, send it to bazel-discuss@googlegroups.com
- if this is a _bug_ or _feature request_, fill the form below as best as you can.
-->

### Description of the problem / feature request:

Repository rules already have access to `repository_os` which allows rules to change their behaviour based on the exec OS. Repository rules should also know the exec architecture. Repository rules currently work around this limitation by  `uname -m`, which isn't ideal. See [rules_go](https://github.com/bazelbuild/rules_go/blob/fd11dd2768669dc2cc1f3a11f2b0b81d84e81c32/go/private/sdk.bzl#L253).

### Feature requests: what underlying problem are you trying to solve with this feature?

Repository rules create potentially brittle and strange workarounds to get the exec CPU. There is no standardisation and no official guidance on how best to create repository rules which target multiple platforms.

### What operating system are you running Bazel on?

Linux x86_64

### What's the output of `bazel info release`?

```sh
❯ bazel info release
Starting local Bazel server and connecting to it...
release 5.0.0
```

###  Have you found anything relevant by searching the web?

> Replace this line with your answer.

- https://github.com/bazelbuild/bazel/issues/11655#issuecomment-734212603
- https://github.com/bazelbuild/bazel/issues/4636 (request was target, not exec)

### Any other information, logs, or outputs that you want to share?

N/A


Issue comments:
I've found the need for this as well, when trying to target x86_64 macOS RBE from arm64 macOS. Repository rules incorrectly provide arm64 binaries which fail to run on RBE.
Also see [rules_rust](https://github.com/bazelbuild/rules_rust/blob/14468f97b9a06ab6b33e94bc87c1d24b2433e5f8/crate_universe/private/util.bzl#L18).
This seems like a reasonable request. It shouldn't be too hard to add a field to `repository_os`, so if anyone wants to send a PR, it would be greatly appreciated.
I started working on this in #14738, but then noticed that this is more subtle than expected: The natural implementation would be to just return `System.getProperty("os.arch").toLowerCase(Locale.ROOT)`, but that is just the architecture of the JDK Bazel is running on, which is not necessarily the same as the architecture of the operating system Bazel is running on. For example, which value should this have in the case of an x86_64 JDK executing Bazel using Rosetta on an aarch64/arm64 M1 Mac?

@uhthomas @brentleyjones Could you clarify what exactly you would want to know the architecture of? You also referred to RBE, but I think that repository rules could only ever provide the _host_ CPU, not the _exec_ CPU (which isn't even known during the loading phase since it depends on the result of toolchain resolution). Is that sufficient for your use cases?
For a M1 mac running through Rosetta x86_64 is right IMO, since the emulation should affect everything. 
The intent of this issue is to solve the multi cpu/os repository problem where rule maintainers are forced to write odd hacks to detect the exec (host?) CPU. For example the Go rules declare a signal repository, `go_sdk`, which will automatically detect the correct SDK to use. The `go_download_sdk()` rule uses [uname](https://github.com/bazelbuild/rules_go/blob/fd11dd2768669dc2cc1f3a11f2b0b81d84e81c32/go/private/sdk.bzl#L253) to do this detection.

It's been suggested that multiple repositories for the corresponding cpu and os should be declared ([ref](https://github.com/bazelbuild/bazel/issues/11655#issuecomment-734212603)), but this can be really messy and weird.

`select` does not work with repository rules and there is no clear guidance on how to actually do this, which is odd considering this is extremely common.

The M1 situation is a bit weird and I don't really know what the right answer is. I think it should depend on whether the architecture of the Bazel binary, as there is no other way to differentiate between native and rosetta at current.

Does what I've written make sense, and is it clear what the goal of this issue is?
>Could you clarify what exactly you would want to know the architecture of? You also referred to RBE, but I think that repository rules could only ever provide the host CPU, not the exec CPU (which isn't even known during the loading phase since it depends on the result of toolchain resolution). Is that sufficient for your use cases?

I would like the know the architecture of `--host_platform`. If I don't run Bazel through Rosetta, all repository rules will see arm64 on an M1 Mac, even if I set `--host_platform` to a platform with `x86_64` cpu.
The architecture of the Bazel binary, at least assuming it uses the shipped JDK, the default `cpu` constraint on the auto-detected host platform and the value of the `os.arch` Java system property are all the same as far as I know, so @uhthomas and @keith requests would be handled by my PR.

@brentleyjones I understand your use case, but this seems difficult to do: Strictly speaking, the platform specified by `--host_platform` is an opaque collection of constraint values, which may theoretically not even contain one for the constraint setting `@platforms//cpu`. And even if we come up with logic for this (admittedly very special) case, I fear getting a constraint value of a platform identified by a label from the context of a repository rule is pretty difficult.
Awesome @fmeum, your PR looks great. Thanks for taking a look at this! :)

Is there some reference which lists the possible values of `os.arch` and `os.name`?

For my own sanity also, are repository rules executed on the host platform, or the execution platform? More specifically, will the values of `os.arch` and `os.name` be of the host platform, or execution platform?
Additionally, how does this all fit into platforms? If at all. I think it might be strange to use the Platforms API in most cases, but the repository_os struct in others. What are your thoughts? 
> Is there some reference which lists the possible values of `os.arch` and `os.name`?

The two sources I usually refer to are the OpenJDK sources and Bazel's [CPU](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/util/CPU.java) and [OS](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/util/OS.java) classes.

> For my own sanity also, are repository rules executed on the host platform, or the execution platform? More specifically, will the values of `os.arch` and `os.name` be of the host platform, or execution platform?

> Additionally, how does this all fit into platforms? If at all. I think it might be strange to use the Platforms API in most cases, but the repository_os struct in others. What are your thoughts?

My understanding is that repository rules are always executed locally (i.e., on the host), even with RBE, and don't really have a concept of platforms. Rather, platforms could be used to select the correct repository among the instances of a repository rule for the different platforms. Bazel would then only fetch the repository rules that are relevant for the current configuration. In this context, `repository_os` is more of a shortcut geared towards local execution, whereas sophisticated RBE setups would probably favor the approach described in https://github.com/bazelbuild/bazel/issues/11655#issuecomment-734212603.

 I think https://github.com/bazelbuild/bazel/pull/14738 solves exactly the request of this issue. The `repository_ctx.os.name` and `repository_ctx.os.arch` return the values for the host platform (which is the one Bazel runs on), this should be enough as in most cases repository rules run locally. 

But we do have a way to make `repository.execute` [run commands on RBE](https://docs.bazel.build/versions/main/skylark/lib/globals.html#repository_rule(implementation,%20attrs,%20local,%20environ,%20configure,%20remotable,%20doc)), but this is a very much experimental feature and currently unmaintained.

I also agree if Bazel runs in a simulation it should just get the simulated arch. We do provide native arm64 binaries for Linux and macOS, arm64 support for Windows has also been [added](https://github.com/bazelbuild/bazel/issues/14339) recently. 
@bazel-io flag
(I miss-read what that command was for, this isn't a blocker, sorry.)
OK, I removed the label. But I guess we could still cherry-pick this to 5.1? @Wyverald 
Yeah, that was what I wanted to do. Per the process, it seems we just open the PR instead of flagging it.
@bazel-io fork 5.1
The fork command will create an associate issue to track the cherry-pick, feel free to send a PR!
Amazing, thank you for the work on this. I look forward to using it! I'll make contributions for the major rules to also take advantage of this new attribute when available.

For clarification; this only provides info for the host OS, correct? What happens in a remote execution context? Are those actions still executed locally? I suppose it depends on the declared toolchains.

Is there any official guidance on how best to write repository rules for compatibility with remote execution? It seems to me like it may still be the case that rule authors must declare every cpu/os combination as repositories and reference them via toolchains.

Maybe I'm missing something 🙂



Pull request review comments:

Patch that resolved the issue:
diff --git a/src/main/java/com/google/devtools/build/lib/bazel/repository/starlark/StarlarkOS.java b/src/main/java/com/google/devtools/build/lib/bazel/repository/starlark/StarlarkOS.java
index dfd8e1209d5956..83df566cabdab9 100644
--- a/src/main/java/com/google/devtools/build/lib/bazel/repository/starlark/StarlarkOS.java
+++ b/src/main/java/com/google/devtools/build/lib/bazel/repository/starlark/StarlarkOS.java
@@ -17,6 +17,7 @@
 import com.google.common.collect.ImmutableMap;
 import com.google.devtools.build.docgen.annot.DocCategory;
 import com.google.devtools.build.lib.concurrent.ThreadSafety.Immutable;
+import java.util.Locale;
 import java.util.Map;
 import net.starlark.java.annot.StarlarkBuiltin;
 import net.starlark.java.annot.StarlarkMethod;
@@ -49,8 +50,20 @@ public ImmutableMap<String, String> getEnvironmentVariables() {
   @StarlarkMethod(
       name = "name",
       structField = true,
-      doc = "A string identifying the current system Bazel is running on.")
+      doc =
+          "A string identifying the operating system Bazel is running on (the value of the"
+              + " \"os.name\" Java property).")
   public String getName() {
-    return System.getProperty("os.name").toLowerCase();
+    return System.getProperty("os.name").toLowerCase(Locale.ROOT);
+  }
+
+  @StarlarkMethod(
+      name = "arch",
+      structField = true,
+      doc =
+          "A string identifying the architecture Bazel is running on (the value of the \"os.arch\""
+              + " Java property).")
+  public String getArch() {
+    return System.getProperty("os.arch").toLowerCase(Locale.ROOT);
   }
 }
diff --git a/tools/cpp/lib_cc_configure.bzl b/tools/cpp/lib_cc_configure.bzl
index cf602ab1b0cb76..f8689fb70b5048 100644
--- a/tools/cpp/lib_cc_configure.bzl
+++ b/tools/cpp/lib_cc_configure.bzl
@@ -179,38 +179,34 @@ def execute(
 
 def get_cpu_value(repository_ctx):
     """Compute the cpu_value based on the OS name. Doesn't %-escape the result!"""
-    os_name = repository_ctx.os.name.lower()
+    os_name = repository_ctx.os.name
+    arch = repository_ctx.os.arch
     if os_name.startswith("mac os"):
         # Check if we are on x86_64 or arm64 and return the corresponding cpu value.
-        result = repository_ctx.execute(["uname", "-m"])
-        return "darwin" + ("_arm64" if result.stdout.strip() == "arm64" else "")
+        return "darwin" + ("_arm64" if arch == "aarch64" else "")
     if os_name.find("freebsd") != -1:
         return "freebsd"
     if os_name.find("openbsd") != -1:
         return "openbsd"
     if os_name.find("windows") != -1:
-        arch = (get_env_var(repository_ctx, "PROCESSOR_ARCHITECTURE", "", False) or
-                get_env_var(repository_ctx, "PROCESSOR_ARCHITEW6432", "", False))
-        if arch == "ARM64":
+        if arch == "aarch64":
             return "arm64_windows"
         else:
             return "x64_windows"
 
-    # Use uname to figure out whether we are on x86_32 or x86_64
-    result = repository_ctx.execute(["uname", "-m"])
-    if result.stdout.strip() in ["power", "ppc64le", "ppc", "ppc64"]:
+    if arch in ["power", "ppc64le", "ppc", "ppc64"]:
         return "ppc"
-    if result.stdout.strip() in ["s390x"]:
+    if arch in ["s390x"]:
         return "s390x"
-    if result.stdout.strip() in ["mips64"]:
+    if arch in ["mips64"]:
         return "mips64"
-    if result.stdout.strip() in ["riscv64"]:
+    if arch in ["riscv64"]:
         return "riscv64"
-    if result.stdout.strip() in ["arm", "armv7l"]:
+    if arch in ["arm", "armv7l"]:
         return "arm"
-    if result.stdout.strip() in ["aarch64"]:
+    if arch in ["aarch64"]:
         return "aarch64"
-    return "k8" if result.stdout.strip() in ["amd64", "x86_64", "x64"] else "piii"
+    return "k8" if arch in ["amd64", "x86_64", "x64"] else "piii"
 
 def is_cc_configure_debug(repository_ctx):
     """Returns True if CC_CONFIGURE_DEBUG is set to 1."""
diff --git a/tools/jdk/local_java_repository.bzl b/tools/jdk/local_java_repository.bzl
index 4c0a433f8f2a1c..12908eda765a84 100644
--- a/tools/jdk/local_java_repository.bzl
+++ b/tools/jdk/local_java_repository.bzl
@@ -132,7 +132,7 @@ def _local_java_repository_impl(repository_ctx):
         "workspace(name = \"{name}\")\n".format(name = repository_ctx.name),
     )
 
-    extension = ".exe" if repository_ctx.os.name.lower().find("windows") != -1 else ""
+    extension = ".exe" if repository_ctx.os.name.find("windows") != -1 else ""
     java_bin = java_home_path.get_child("bin").get_child("java" + extension)
 
     if not java_bin.exists:
diff --git a/tools/osx/xcode_configure.bzl b/tools/osx/xcode_configure.bzl
index 2b819f07ec9f9a..94bd896c4a11aa 100644
--- a/tools/osx/xcode_configure.bzl
+++ b/tools/osx/xcode_configure.bzl
@@ -271,7 +271,7 @@ def _impl(repository_ctx):
       repository_ctx: The repository context.
     """
 
-    os_name = repository_ctx.os.name.lower()
+    os_name = repository_ctx.os.name
     build_contents = "package(default_visibility = ['//visibility:public'])\n\n"
     if (os_name.startswith("mac os")):
         build_contents += _darwin_build_file(repository_ctx)


Test patch: