Repository: minio/minio (language: go)

Issue:
Unable to upgrade beyond RELEASE.2024-07-16T23-46-41Z due to 403 responses
A cluster of mine is currently on `RELEASE.2024-07-16T23-46-41Z` and I wanted to upgrade to latest, however the first node that gets restarted never becomes healthy because it just keeps receiving 403 responses from peers. The smallest step to `RELEASE.2024-07-26T20-48-21Z` also already triggers this issue. I have the suspicion that this might very well be related to minio/operator#2229 which fails to perform the expected simultaneous restart of all peers. A rollback to the previous version works fine.

## Expected Behavior

Upgrade works.

## Current Behavior

Rolling upgrade fails due to 403 responses from peers.

## Possible Solution

I suspect that deleting all minio pods at the same time might do the trick, even though it will take the cluster offline for a second.

## Steps to Reproduce (for bugs)

1. Deploy a cluster with operator 6.0.2 on version `RELEASE.2024-07-16T23-46-41Z`
2. Attempt upgrade to `RELEASE.2024-07-26T20-48-21Z`

## Context

The k8s cluster runs on k0s `v1.30.2+k0s` with minio operator chart 6.0.1 and operator image version 6.0.2.




Discussion / hints:
@pschichtel can you share the logs from the node that shows 403 responses? 
here you go:

<details>

```
INFO:
 You are running an older version of MinIO released 1 week before the latest release
 Update: Run `mc admin update ALIAS`
API: SYSTEM.grid
Time: 18:54:35 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-1.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.685s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
INFO: Waiting for at least 2 remote servers with valid configuration to be online
INFO: Following servers are currently offline or unreachable [https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-3.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-2.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-0.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-1.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected]
API: SYSTEM.grid
Time: 18:54:35 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-0.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.288s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
INFO: Waiting for at least 2 remote servers with valid configuration to be online
INFO: Following servers are currently offline or unreachable [https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-3.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-2.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-1.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-0.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected]
API: SYSTEM.grid
Time: 18:54:36 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-0.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.303s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
API: SYSTEM.grid
Time: 18:54:36 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-1.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.616s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
INFO: Waiting for at least 2 remote servers with valid configuration to be online
INFO: Following servers are currently offline or unreachable [https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-3.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-0.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-1.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-2.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected]
API: SYSTEM.grid
Time: 18:54:38 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-0.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.737s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
API: SYSTEM.grid
Time: 18:54:38 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-1.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.961s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
INFO: Waiting for at least 2 remote servers with valid configuration to be online
INFO: Following servers are currently offline or unreachable [https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-3.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-2.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-0.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-1.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected]
API: SYSTEM.grid
Time: 18:54:39 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-0.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.387s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
API: SYSTEM.grid
Time: 18:54:40 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-1.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.946s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
INFO: Waiting for at least 2 remote servers with valid configuration to be online
INFO: Following servers are currently offline or unreachable [https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-3.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-2.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-1.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-0.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected]
API: SYSTEM.grid
Time: 18:54:41 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-0.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.295s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
API: SYSTEM.grid
Time: 18:54:42 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-1.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.561s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
API: SYSTEM.grid
Time: 18:54:42 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-0.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.016s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
INFO: Waiting for at least 2 remote servers with valid configuration to be online
INFO: Following servers are currently offline or unreachable [https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-3.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-1.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-0.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-2.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected]
API: SYSTEM.grid
Time: 18:54:43 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-0.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.637s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
API: SYSTEM.grid
Time: 18:54:43 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-1.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.858s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
INFO: Waiting for at least 2 remote servers with valid configuration to be online
INFO: Following servers are currently offline or unreachable [https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-3.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-0.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-2.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-1.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected]
API: SYSTEM.grid
Time: 18:54:45 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-0.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.578s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
INFO: Waiting for at least 2 remote servers with valid configuration to be online
INFO: Following servers are currently offline or unreachable [https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-3.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-2.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-1.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-0.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected]
API: SYSTEM.grid
Time: 18:54:45 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-1.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.507s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
API: SYSTEM.grid
Time: 18:54:46 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-0.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.061s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
INFO: Waiting for at least 2 remote servers with valid configuration to be online
INFO: Following servers are currently offline or unreachable [https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-3.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-2.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-0.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-1.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected]
API: SYSTEM.grid
Time: 18:54:47 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-1.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.468s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
API: SYSTEM.grid
Time: 18:54:47 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-0.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.557s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
INFO: Waiting for at least 2 remote servers with valid configuration to be online
INFO: Following servers are currently offline or unreachable [https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-3.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-1.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-0.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-2.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected]
API: SYSTEM.grid
Time: 18:54:48 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-1.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.944s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
API: SYSTEM.grid
Time: 18:54:49 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-0.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.905s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
INFO: Waiting for at least 2 remote servers with valid configuration to be online
INFO: Following servers are currently offline or unreachable [https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-3.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-1.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-2.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-0.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected]
API: SYSTEM.grid
Time: 18:54:50 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-1.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.258s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
API: SYSTEM.grid
Time: 18:54:51 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-0.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.326s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
INFO: Waiting for at least 2 remote servers with valid configuration to be online
INFO: Following servers are currently offline or unreachable [https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-3.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-1.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-2.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-0.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected]
API: SYSTEM.grid
Time: 18:54:51 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-1.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.921s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
API: SYSTEM.grid
Time: 18:54:52 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-0.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.506s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
INFO: Waiting for at least 2 remote servers with valid configuration to be online
INFO: Following servers are currently offline or unreachable [https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-3.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-0.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-1.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-2.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected]
API: SYSTEM.grid
Time: 18:54:53 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-1.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.786s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
API: SYSTEM.grid
Time: 18:54:54 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-0.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.903s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
INFO: Waiting for at least 2 remote servers with valid configuration to be online
INFO: Following servers are currently offline or unreachable [https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-2.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-0.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-3.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-1.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected]
API: SYSTEM.grid
Time: 18:54:55 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-1.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.841s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
API: SYSTEM.grid
Time: 18:54:56 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-0.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.948s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
INFO: Waiting for at least 2 remote servers with valid configuration to be online
INFO: Following servers are currently offline or unreachable [https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-3.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-0.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-1.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-2.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected]
API: SYSTEM.grid
Time: 18:54:57 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-1.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.541s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
INFO: Waiting for at least 2 remote servers with valid configuration to be online
INFO: Following servers are currently offline or unreachable [https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-3.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-1.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-2.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-0.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected]
API: SYSTEM.grid
Time: 18:54:57 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-0.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.494s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
API: SYSTEM.grid
Time: 18:54:59 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-1.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.942s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
INFO: Waiting for at least 2 remote servers with valid configuration to be online
INFO: Following servers are currently offline or unreachable [https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-3.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-0.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-1.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-2.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected]
API: SYSTEM.grid
Time: 18:54:59 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-0.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.721s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
INFO: Waiting for at least 2 remote servers with valid configuration to be online
INFO: Following servers are currently offline or unreachable [https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-3.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-1.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-2.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-0.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected]
API: github.com/minio/minio/cmd.registerDistErasureRouters.(*Manager).Handler.func1
Time: 18:55:00 UTC 08/12/2024
RequestID: 17EB1006231800BF
RemoteHost: 0.0.0.161
Host: node-4.tenant-hl.namespace.svc.cluster.local:9000
Error: auth 0.0.0.161:50176: Malformed authentication input (*fmt.wrapError)
       5: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       4: internal/logger/logonce.go:149:logger.LogOnceIf()
       3: internal/grid/connection.go:59:grid.gridLogOnceIf()
       2: internal/grid/manager.go:163:cmd.registerDistErasureRouters.(*Manager).Handler.func1()
       1: net/http/server.go:2171:http.HandlerFunc.ServeHTTP()
API: SYSTEM.grid
Time: 18:55:00 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-1.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.644s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
API: SYSTEM.grid
Time: 18:55:01 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-0.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.463s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
INFO: Waiting for at least 2 remote servers with valid configuration to be online
INFO: Following servers are currently offline or unreachable [https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-3.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-0.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-1.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-2.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected]
API: github.com/minio/minio/cmd.registerDistErasureRouters.(*Manager).Handler.func1
Time: 18:55:02 UTC 08/12/2024
RequestID: 17EB1006621F67B8
RemoteHost: 0.0.0.161
Host: node-4.tenant-hl.namespace.svc.cluster.local:9000
Error: auth 0.0.0.161:50190: Malformed authentication input (*fmt.wrapError)
       5: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       4: internal/logger/logonce.go:149:logger.LogOnceIf()
       3: internal/grid/connection.go:59:grid.gridLogOnceIf()
       2: internal/grid/manager.go:163:cmd.registerDistErasureRouters.(*Manager).Handler.func1()
       1: net/http/server.go:2171:http.HandlerFunc.ServeHTTP()
API: SYSTEM.grid
Time: 18:55:02 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-1.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.821s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
API: SYSTEM.grid
Time: 18:55:02 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-0.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.862s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
API: github.com/minio/minio/cmd.registerDistErasureRouters.(*Manager).Handler.func1
Time: 18:55:03 UTC 08/12/2024
RequestID: 17EB1006AFD877A2
RemoteHost: 0.0.0.161
Host: node-4.tenant-hl.namespace.svc.cluster.local:9000
Error: auth 0.0.0.161:50196: Malformed authentication input (*fmt.wrapError)
       5: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       4: internal/logger/logonce.go:149:logger.LogOnceIf()
       3: internal/grid/connection.go:59:grid.gridLogOnceIf()
       2: internal/grid/manager.go:163:cmd.registerDistErasureRouters.(*Manager).Handler.func1()
       1: net/http/server.go:2171:http.HandlerFunc.ServeHTTP()
INFO: Waiting for at least 2 remote servers with valid configuration to be online
INFO: Following servers are currently offline or unreachable [https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-3.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-0.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-2.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-1.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected]
API: github.com/minio/minio/cmd.registerDistErasureRouters.(*Manager).Handler.func1
Time: 18:55:03 UTC 08/12/2024
RequestID: 17EB1006C4A8DE9F
RemoteHost: 0.0.0.28
Host: node-4.tenant-hl.namespace.svc.cluster.local:9000
Error: auth 0.0.0.28:54806: Malformed authentication input (*fmt.wrapError)
       5: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       4: internal/logger/logonce.go:149:logger.LogOnceIf()
       3: internal/grid/connection.go:59:grid.gridLogOnceIf()
       2: internal/grid/manager.go:163:cmd.registerDistErasureRouters.(*Manager).Handler.func1()
       1: net/http/server.go:2171:http.HandlerFunc.ServeHTTP()
API: SYSTEM.grid
Time: 18:55:04 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-1.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.902s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
API: SYSTEM.grid
Time: 18:55:04 UTC 08/12/2024
Error: grid: https://node-4.tenant-hl.namespace.svc.cluster.local:9000 re-connecting to https://node-0.tenant-hl.namespace.svc.cluster.local:9000: unexpected HTTP response status: 403 (ws.StatusError) Sleeping 1.945s (3) (*fmt.wrapError)
       6: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       5: internal/logger/logonce.go:149:logger.LogOnceIf()
       4: internal/grid/connection.go:59:grid.gridLogOnceIf()
       3: internal/grid/connection.go:672:grid.(*Connection).connect.func1()
       2: internal/grid/connection.go:678:grid.(*Connection).connect()
       1: internal/grid/connection.go:275:grid.newConnection.func3()
API: github.com/minio/minio/cmd.registerDistErasureRouters.(*Manager).Handler.func1
Time: 18:55:04 UTC 08/12/2024
RequestID: 17EB1006F8220F15
RemoteHost: 0.0.0.161
Host: node-4.tenant-hl.namespace.svc.cluster.local:9000
Error: auth 0.0.0.161:54648: Malformed authentication input (*fmt.wrapError)
       5: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       4: internal/logger/logonce.go:149:logger.LogOnceIf()
       3: internal/grid/connection.go:59:grid.gridLogOnceIf()
       2: internal/grid/manager.go:163:cmd.registerDistErasureRouters.(*Manager).Handler.func1()
       1: net/http/server.go:2171:http.HandlerFunc.ServeHTTP()
INFO: Waiting for at least 2 remote servers with valid configuration to be online
INFO: Following servers are currently offline or unreachable [https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-3.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-1.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-0.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected https://node-4.tenant-hl.namespace.svc.cluster.local:9000->https://node-2.tenant-hl.namespace.svc.cluster.local:9000 is unreachable: remote disconnected]
API: github.com/minio/minio/cmd.registerDistErasureRouters.(*Manager).Handler.func1
Time: 18:55:05 UTC 08/12/2024
RequestID: 17EB100726FF0828
RemoteHost: 0.0.0.28
Host: node-4.tenant-hl.namespace.svc.cluster.local:9000
Error: auth 0.0.0.28:54822: Malformed authentication input (*fmt.wrapError)
       5: internal/logger/logonce.go:118:logger.(*logOnceType).logOnceIf()
       4: internal/logger/logonce.go:149:logger.LogOnceIf()
       3: internal/grid/connection.go:59:grid.gridLogOnceIf()
       2: internal/grid/manager.go:163:cmd.registerDistErasureRouters.(*Manager).Handler.func1()
       1: net/http/server.go:2171:http.HandlerFunc.ServeHTTP()
```

</detail>
```
Host: node-4.tenant-hl.namespace.svc.cluster.local:9000
Error: auth 0.0.0.28:54822: Malformed authentication input (*fmt.wrapError)
```

This is likely due to the need for an operator to restart the nodes. This would require manual intervention of restarting the pods via killing all of them. 
I was wondering: would the `mcli admin update` followed by the image upgrade via operator work?
> I was wondering: would the `mcli admin update` followed by the image upgrade via operator work?

Yes it would work I guess, but it will download the binary from our public repository. 


Actually... I can reproduce this with a single-node docker setup as well. The only difference is: Minio actually starts, but I can't login using the root user (`invalid Login` response).

These single-node development instances also use the root user in the application with the S3 SDK, which leads to these exceptions:

```
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: The Access Key Id you provided does not exist in our records. (Service: S3, Status Code: 403, Request ID: 17EB56B412051AF3, Extended Request ID: dd9025bab4ad464b049177c95eb6ebf374d3b3fd1af9251148b658df7ac2e3e8)
     at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
     at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:108)
     at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:85)
     at software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:43)
     at software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$7(BaseClientHandler.java:279)
     at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:92)
     at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1150)
     at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510)
     at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179)
     at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onComplete(AsyncResponseHandler.java:135)
     at software.amazon.awssdk.core.internal.metrics.BytesReadTrackingPublisher$BytesReadTracker.onComplete(BytesReadTrackingPublisher.java:74)
     at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler$DataCountingPublisher$1.onComplete(ResponseHandler.java:519)
     at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.runAndLogError(ResponseHandler.java:254)
     at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.access$600(ResponseHandler.java:77)
     at software.amazon.awssdk.http.nio.netty.internal.ResponseHandler$PublisherAdapter$1.onComplete(ResponseHandler.java:375)
     at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.publishMessage(HandlerPublisher.java:402)
     at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.flushBuffer(HandlerPublisher.java:338)
     at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.receivedDemand(HandlerPublisher.java:291)
     at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher.access$200(HandlerPublisher.java:61)
     at software.amazon.awssdk.http.nio.netty.internal.nrs.HandlerPublisher$ChannelSubscription$1.run(HandlerPublisher.java:495)
     at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
     at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
     at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
     at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994)
     at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
     at java.base/java.lang.Thread.run(Thread.java:1583)
```

The minio container doesn't log anything beyond the usual startup logs.

This is a docker-swarm stack reduced from our deployment in these environments:

<details>

```yaml
services:
  minio:
    image: "docker.io/minio/minio:RELEASE.2024-08-03T04-33-23Z@sha256:c933bb53ac226d1f4bdfeac66a3854d02903771e91d7ee2cbc330841a1d77d7e"
    hostname: "{{ .Service.Name }}-{{ .Task.Slot }}"
    deploy:
      mode: "replicated"
      replicas: 1
      restart_policy:
        condition: "on-failure"
        delay: 10s
      update_config:
        parallelism: 1
        delay: 10s
        monitor: 1m
        failure_action: rollback
        order: "stop-first"
      rollback_config:
        parallelism: 1
        delay: 10s
        monitor: 1m
        failure_action: pause
        order: "stop-first"
    command:
      - "server"
      - "--certs-dir"
      - "/run/secrets"
      - "--console-address"
      - ":9443"
      - "--json"
      - "/data"
    environment:
      MINIO_BROWSER_REDIRECT_URL: "https://console.example.org"
      MINIO_KMS_SECRET_KEY_FILE: "kms-key"
      MINIO_PROMETHEUS_AUTH_TYPE: "public"
      MINIO_ROOT_PASSWORD_FILE: "minio_root_password"
      MINIO_ROOT_USER_FILE: "minio_root_user"
      MINIO_SERVER_URL: "https://example.org"
    secrets:
      - source: "minio_kms_key.3353219364"
        target: "kms-key"
      - source: "minio_root_password.652613740"
        target: "minio_root_password"
      - source: "minio_root_user.100729354"
        target: "minio_root_user"
      - source: "minio_tls_private_key.295846674"
        target: "private.key"
      - source: "minio_tls_public_crt.4109694386"
        target: "public.crt"
      - source: "traefik-tls-certificate.3416996646"
        target: "CAs/ca.crt"
    volumes:
      - "minio_data:/data"
    networks:
      - "application"

secrets:
  minio_kms_key.3353219364:
    file: ./infrastructure/minio/kms-key.txt
  minio_root_password.652613740:
    file: /tmp/tmp.lNL8F4q4EA
  minio_root_user.100729354:
    file: ./infrastructure/minio/root-user.txt
  minio_tls_private_key.295846674:
    file: ./infrastructure/minio/tls-private.key
  minio_tls_public_crt.4109694386:
    file: ./infrastructure/minio/tls-public.crt

volumes:
  minio_data:

networks:
  application:
```

</details>

These are all development environment which I can freely change around if necessary.
is this STS credentials that you are using? @pschichtel 
> Actually... I can reproduce this with a single-node docker setup as well. The only difference is: Minio actually starts, but I can't login using the root user (`invalid Login` response).
> 
> These single-node development instances also use the root user in the application with the S3 SDK, which leads to these exceptions:

This doesn't make any sense, we have customers deploying always and upgrading. You might have changed something here.
> is this STS credentials that you are using?

In the docker swarm setup? It's just the root user, no other credentials exist.

> This doesn't make any sense, we have customers deploying always and upgrading. You might have changed something here.

This has independently happened to at least two developers with separate environments and on a shared test environment, which is deployed automatically gitops style. The only thing that changed is the container version. 
can you provide a reproducer for it @pschichtel ?
I'll look into that. In the meantime: `RELEASE.2024-07-26T20-48-21Z' introduced https://github.com/minio/minio/pull/20122, which changes something around static KMS keys, which we use in our docker swarm development environments. All our development environments break once we upgrade to that version or newer and all of them use static KMS keys. I'd assume, if the KMS key is not properly loaded, then it also won't be able to decrypt the root credentials.

So my assumption is: Start a docker container with static KMS configured like in my docker stack on version `RELEASE.2024-07-16T23-46-41Z`, login (that should work), then upgrade to `RELEASE.2024-07-26T20-48-21Z` and try to login again. probably doesn't work.
Surprisingly, if I run minio like this without actually mounting the secrets, then it goes on without complaining: `docker run --rm -it -e MINIO_ROOT_USER_FILE=minio_root_user -e MINIO_ROOT_PASSWORD_FILE=minio_root_password -e MINIO_KMS_SECRET_KEY_FILE=kms-key docker.io/minio/minio:RELEASE.2024-07-16T23-46-41Z server --certs-dir /run/secrets --console-address :9443 --json /data`.

I'm pretty sure that was not the case in the past.
I just removed minio's data and started on a fresh volume. the upgrade works, but also KMS doesn't seem to be configured. 
I don't have an explanation for why login works here after the upgrade, but files uploaded with KMS-S3 before the upgrade cannot be accessed after the upgrade, because "KMS not configured".
@harshavardhana Yes. [your commit](https://github.com/harshavardhana/minio/commit/7eeedd4916cb53c6d44f2b7009f0015e350e403a) was indeed a breaking change, as it now doesn't work to specify just the secret name. I migrated all env vars over to full file paths (makes more sense anyway). But this whole problem was unrelated to this issue.
> @harshavardhana Yes. [your commit](https://github.com/harshavardhana/minio/commit/7eeedd4916cb53c6d44f2b7009f0015e350e403a) was indeed a breaking change, as it now doesn't work to specify just the secret name. I migrated all env vars over to full file paths (makes more sense anyway). But this whole problem was unrelated to this issue.

Well if i don't do it there is no other easier way for us to manage the secret file thing.

It had to removed to fix the original reported problem.
I guess at this point it might make sense to drop the functionality for all env vars given that it is now inconsistent already. It's oddly docker-specific anyway and providing full paths works fine on docker too.
> I guess at this point it might make sense to drop the functionality for all env vars given that it is now inconsistent already. It's oddly docker-specific anyway and providing full paths works fine on docker too.

it can't be removed as there are two types of users one with full path, and one with relative path. I can attempt to do it automatically via `/run/secrets` as well just in case for other type of relative path users. 


Fix patch:
diff --git a/internal/kms/config.go b/internal/kms/config.go
index 2a50ca3b7b4bf..7d5952764b0ac 100644
--- a/internal/kms/config.go
+++ b/internal/kms/config.go
@@ -26,6 +26,7 @@ import (
 	"errors"
 	"fmt"
 	"os"
+	"path/filepath"
 	"strings"
 	"sync/atomic"
 	"syscall"
@@ -254,6 +255,13 @@ func Connect(ctx context.Context, opts *ConnectionOptions) (*KMS, error) {
 		var s string
 		if lookup(EnvKMSSecretKeyFile) {
 			b, err := os.ReadFile(env.Get(EnvKMSSecretKeyFile, ""))
+			if err != nil && !os.IsNotExist(err) {
+				return nil, err
+			}
+			if os.IsNotExist(err) {
+				// Relative path where "/run/secrets" is the default docker path for secrets
+				b, err = os.ReadFile(filepath.Join("/run/secrets", env.Get(EnvKMSSecretKeyFile, "")))
+			}
 			if err != nil {
 				return nil, err
 			}


Test patch: