Repository: TraceMachina/nativelink (language: rust)

Issue:
`docker-compose` Example does not appear to work
**Expected behaviour**: I can run the `docker-compose` example and get a basic remote executor
**Actual behaviour**: I run the `docker-compose` example and get a remote cache, but not a remote executor

**More details**

**Operating System**: Fedora 40
**Nativelink version**: cloned from repository @ `main`
**Bazel version**: 7.4.0

**Steps to reproduce:**

First, I clone the main `nativelink` repository and run the `docker-compose` example:

```
$ git clone https://github.com/TraceMachina/nativelink.git 
$ cd nativelink
$ cd deployment-examples/docker-compose
$ docker compose up
...
[+] Running 3/0
 ✔ Container docker-compose-nativelink_executor-1   Created                                                                                                                                                                                           0.0s 
 ✔ Container docker-compose-nativelink_local_cas-1  Created                                                                                                                                                                                           0.0s 
 ✔ Container docker-compose-nativelink_scheduler-1  Created    
...
nativelink_executor-1   |   2024-11-19T10:05:42.072224Z  WARN nativelink_worker::local_worker: Worker registered with scheduler, worker_id: bb31df10-6cd3-42ca-a3e8-179d0a5b91a8
nativelink_executor-1   |     at nativelink-worker/src/local_worker.rs:571
nativelink_executor-1   |     in nativelink::worker with name: "worker_0"
nativelink_executor-1   | 
```

Then, in a Bazel project, I add:
```
build --remote_cache=grpc://127.0.0.1:50051
build --remote_executor=grpc://127.0.0.1:50052
build --remote_default_exec_properties=cpu_count=1
```

I then run `bazel info` and get the following error:
```
ERROR: Failed to query remote execution capabilities: Remote execution is not supported by the remote server, or the current account is not authorized to use remote execution.
java.io.IOException: Remote execution is not supported by the remote server, or the current account is not authorized to use remote execution.
        at com.google.devtools.build.lib.remote.GoogleChannelConnectionFactory.lambda$getAndVerifyServerCapabilities$4(GoogleChannelConnectionFactory.java:140)
```

and the `docker compose` log shows:
```
nativelink_local_cas-1  |   2024-11-19T10:05:59.839600Z ERROR nativelink::services: Failed running service, err: hyper::Error(Io, Kind(NotConnected))
nativelink_local_cas-1  |     at src/bin/nativelink.rs:855
nativelink_local_cas-1  |     in nativelink::services::http_connection with remote_addr: xxx.xxx.xxx.xxx:41694, socket_addr: 0.0.0.0:50051
```

I've tried changing the Rust logging level from `warn` to `debug` and then when I run `bazel info` I see:

```
nativelink_scheduler-1  |   2024-11-19T10:12:27.724568Z  INFO nativelink_service::capabilities_server: return: Response { metadata: MetadataMap { headers: {} }, message: ServerCapabilities { cache_capabilities: Some(CacheCapabilities { digest_functions: [Sha256, Blake3], action_cache_update_capabilities: Some(ActionCacheUpdateCapabilities { update_enabled: true }), cache_priority_capabilities: None, max_batch_total_size_bytes: 65536, symlink_absolute_path_strategy: Disallowed, supported_compressors: [], supported_batch_update_compressors: [] }), execution_capabilities: None, deprecated_api_version: None, low_api_version: Some(SemVer { major: 2, minor: 0, patch: 0, prerelease: "" }), high_api_version: Some(SemVer { major: 2, minor: 3, patch: 0, prerelease: "" }) }, extensions: Extensions }
nativelink_scheduler-1  |     at nativelink-service/src/capabilities_server.rs:94
nativelink_local_cas-1  |   2024-11-19T10:12:27.724570Z  INFO nativelink_service::capabilities_server: return: Response { metadata: MetadataMap { headers: {} }, message: ServerCapabilities { cache_capabilities: Some(CacheCapabilities { digest_functions: [Sha256, Blake3], action_cache_update_capabilities: Some(ActionCacheUpdateCapabilities { update_enabled: true }), cache_priority_capabilities: None, max_batch_total_size_bytes: 65536, symlink_absolute_path_strategy: Disallowed, supported_compressors: [], supported_batch_update_compressors: [] }), execution_capabilities: None, deprecated_api_version: None, low_api_version: Some(SemVer { major: 2, minor: 0, patch: 0, prerelease: "" }), high_api_version: Some(SemVer { major: 2, minor: 3, patch: 0, prerelease: "" }) }, extensions: Extensions }
```
... specifically: `execution_capabilities: None`.

Additionally, if I remove the `build --remote_executor=grpc://127.0.0.1:50052` line and try to build, the console log is flooded with the following errors:

```
nativelink_local_cas-1  |   2024-11-19T10:15:17.109353Z ERROR nativelink_service::ac_server: return: Err(Error { code: Internal, messages: ["'instance_name' not configured for ''"] })
```

I'm assuming that there's something wrong with the default example configuration, but I've yet to be able to narrow down exactly what it is.


Discussion / hints:
@swarren12 thank you for reporting the issue. We are reading the thorough write up which we appreciate and will look to respond as soon as possible. 


Two questions I have right away are:

What are the specs of the machine on which you are running this test? OS, version, Docker version, Chip, etc. 

Why did you attempt to run Docker Compose on main rather than a released version? I'm expecting it to be a bit unstable as we only guarantee stability of releases. 
@swarren12 Could you try running the commands with an additional `--remote_instance_name=main` flag? The instance name is configured here (and again a few lines lower): https://github.com/TraceMachina/nativelink/blob/481226be52a84ad5a6b990cc48e9f97512d8ccd2/deployment-examples/docker-compose/scheduler.json#L43

If you don't set the `--remote_instance_name` flag, it expects an empty string (i.e. you could also use `""` instead of `"main"` in the configs if you don't want to use the flag).

@swarren12 If this solves your issue, I think we should remove the "main" from the configs and actually use empty strings by default as this has been a point of confusion in the past. Would you be interested in sending a PR for this?
> @swarren12 Could you try running the commands with an additional `--remote_instance_name=main` flag? The instance name is configured here (and again a few lines lower):
> 
> https://github.com/TraceMachina/nativelink/blob/481226be52a84ad5a6b990cc48e9f97512d8ccd2/deployment-examples/docker-compose/scheduler.json#L43
> 
> If you don't set the `--remote_instance_name` flag, it expects an empty string (i.e. you could also use `""` instead of `"main"` in the configs if you don't want to use the flag).
> 
> @swarren12 If this solves your issue, I think we should remove the "main" from the configs and actually use empty strings by default as this has been a point of confusion in the past. Would you be interested in sending a PR for this?

This appears to work. I'll run a few more tests later today and look at raising a PR if everything looks good.

Thanks!
@aaronmondal I noticed that the other examples (K8s, Chromium) are located under `deploy/...-example`, whilst `docker-compose` is `deployment-example/docker-compose`. Do you think it should be moved to `deploy/docker-compose-example`?

Fix patch:
diff --git a/.bazelrc b/.bazelrc
index 40655abda..ced67a845 100644
--- a/.bazelrc
+++ b/.bazelrc
@@ -33,11 +33,9 @@ build --test_output=errors
 
 build:debug -c dbg
 
-build:self_test --remote_instance_name=main
 build:self_test --remote_cache=grpc://127.0.0.1:50051
 
 build:self_execute --remote_executor=grpc://127.0.0.1:50052
-build:self_execute --remote_instance_name=main
 build:self_execute --remote_default_exec_properties=cpu_count=1
 build:self_execute --platform_suffix=self-execute
 
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index f2448dd7c..ff698c82c 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -159,7 +159,6 @@ jobs:
         docker run --rm --net=host -w /root/nativelink -v $PWD:/root/nativelink trace_machina/nativelink:builder sh -c ' \
           bazel clean && \
           bazel test //... \
-          --remote_instance_name=main \
           --remote_cache=grpc://127.0.0.1:50051 \
           --remote_executor=grpc://127.0.0.1:50052 \
           --remote_default_exec_properties=cpu_count=1 \
@@ -167,7 +166,6 @@ jobs:
         docker run --rm --net=host -w /root/nativelink -v $PWD:/root/nativelink trace_machina/nativelink:builder sh -c ' \
           bazel clean && \
           bazel test //... \
-          --remote_instance_name=main \
           --remote_cache=grpc://127.0.0.1:50051 \
           --remote_executor=grpc://127.0.0.1:50052 \
           --remote_default_exec_properties=cpu_count=1 \
diff --git a/deployment-examples/docker-compose/README.md b/deployment-examples/docker-compose/README.md
index 4ed93cb8f..775c470c6 100644
--- a/deployment-examples/docker-compose/README.md
+++ b/deployment-examples/docker-compose/README.md
@@ -24,7 +24,6 @@ something like:
 
 ```sh
 bazel test //... \
-  --remote_instance_name=main \
   --remote_cache=grpc://127.0.0.1:50051 \
   --remote_executor=grpc://127.0.0.1:50052 \
   --remote_default_exec_properties=cpu_count=1
diff --git a/deployment-examples/docker-compose/local-storage-cas.json b/deployment-examples/docker-compose/local-storage-cas.json
index e7b622043..b847ec2ac 100644
--- a/deployment-examples/docker-compose/local-storage-cas.json
+++ b/deployment-examples/docker-compose/local-storage-cas.json
@@ -41,19 +41,19 @@
     },
     "services": {
       "cas": {
-        "main": {
+        "": {
           "cas_store": "CAS_MAIN_STORE"
         }
       },
       "ac": {
-        "main": {
+        "": {
           "ac_store": "AC_MAIN_STORE"
         }
       },
       "capabilities": {},
       "bytestream": {
         "cas_stores": {
-          "main": "CAS_MAIN_STORE",
+          "": "CAS_MAIN_STORE",
         }
       }
     }
@@ -80,21 +80,26 @@
         }
       }
     },
+    // Each of the services below is keyed on an instance_name.
+    // By default, an empty string is used, but if a different
+    // instance_name is provided, the instance name must be
+    // specified by the invoking tool (e.g. using
+    // `--remote_instance_name=` for Bazel)
     "services": {
       "cas": {
-        "main": {
+        "": {
           "cas_store": "CAS_MAIN_STORE"
         }
       },
       "ac": {
-        "main": {
+        "": {
           "ac_store": "AC_MAIN_STORE"
         }
       },
       "capabilities": {},
       "bytestream": {
         "cas_stores": {
-          "main": "CAS_MAIN_STORE",
+          "": "CAS_MAIN_STORE",
         }
       },
       "health": {},
diff --git a/deployment-examples/docker-compose/scheduler.json b/deployment-examples/docker-compose/scheduler.json
index 4af051738..02eebb31f 100644
--- a/deployment-examples/docker-compose/scheduler.json
+++ b/deployment-examples/docker-compose/scheduler.json
@@ -3,7 +3,7 @@
     "GRPC_LOCAL_STORE": {
       // Note: This file is used to test GRPC store.
       "grpc": {
-        "instance_name": "main",
+        "instance_name": "",
         "endpoints": [
           {"address": "grpc://${CAS_ENDPOINT:-127.0.0.1}:50051"}
         ],
@@ -13,7 +13,7 @@
     "GRPC_LOCAL_AC_STORE": {
       // Note: This file is used to test GRPC store.
       "grpc": {
-        "instance_name": "main",
+        "instance_name": "",
         "endpoints": [
           {"address": "grpc://${CAS_ENDPOINT:-127.0.0.1}:50051"}
         ],
@@ -38,20 +38,25 @@
         "socket_address": "0.0.0.0:50052",
       }
     },
+    // Each of the services below is keyed on an instance_name.
+    // By default, an empty string is used, but if a different
+    // instance_name is provided, the instance name must be
+    // specified by the invoking tool (e.g. using
+    // `--remote_instance_name=` for Bazel)
     "services": {
       "ac": {
-        "main": {
+        "": {
           "ac_store": "GRPC_LOCAL_AC_STORE"
         }
       },
       "execution": {
-        "main": {
+        "": {
           "cas_store": "GRPC_LOCAL_STORE",
           "scheduler": "MAIN_SCHEDULER",
         }
       },
       "capabilities": {
-        "main": {
+        "": {
           "remote_execution": {
             "scheduler": "MAIN_SCHEDULER",
           }
diff --git a/deployment-examples/docker-compose/worker.json b/deployment-examples/docker-compose/worker.json
index 186722e01..08ee5d2bb 100644
--- a/deployment-examples/docker-compose/worker.json
+++ b/deployment-examples/docker-compose/worker.json
@@ -3,7 +3,7 @@
     "GRPC_LOCAL_STORE": {
       // Note: This file is used to test GRPC store.
       "grpc": {
-        "instance_name": "main",
+        "instance_name": "",
         "endpoints": [
           {"address": "grpc://${CAS_ENDPOINT:-127.0.0.1}:50051"}
         ],
@@ -13,7 +13,7 @@
     "GRPC_LOCAL_AC_STORE": {
       // Note: This file is used to test GRPC store.
       "grpc": {
-        "instance_name": "main",
+        "instance_name": "",
         "endpoints": [
           {"address": "grpc://${CAS_ENDPOINT:-127.0.0.1}:50051"}
         ],


Test patch: